Apple Intelligence and More from WWDC
Hey folks,
Usually, we don't do two issues in one day, but because of WWDC, I wanted to get everyone the non-WWDC news first.
Also, IT’S FREE PREVIEW WEEK! All subscribers get every issue every day to see what the whole week looks like for paid subscribers. Enjoy!
OK, so here's my take on all the WWDC announcements. Of course, stay tuned to DTNS for more.
Down below are all my notes for the entire announcement, but here's my non-AI-generated summary of what I think is important.
Apple Intelligence
There are three ways your data can be used to provide a generative feature:
On-device semantic index.
Private Cloud Compute to Apple M1 servers inspected by independent experts for privacy. Data is not stored or exposed to Apple.
You agree with a single request to send specified data to a third party. ChatGPT is the only one at the moment, but more are coming.
What you get:
Siri: natural language and context-aware
Rewrite: can proofread, summarize, etc.
Notification prioritization
Genmoji and Image Playground: Animation, illustration, or sketch
Apple is going to have to answer a lot of questions about privacy, and a lot of folks are not going to trust this because it has AI in the name and especially because ChatGPT is involved. My take is they did a good job at limiting exposure, and all of the questions are answerable. It will be annoying to have to say yes to every ChatGPT integration, but it does make sure you can avoid sharing any data with OpenAI if you want. The Applets' cloud data processing is a little murkier. Granted it's all Apple-controlled, not another company, but there are a lot of questions. They did the right thing in requiring independent vetting of their own processes and any app that wants to interact with them.
Mac OS Sequoia
iPhone mirroring lets you control a locked phone in standby mode to do anything you would do on the iPhone without having to unlock it.
Automatic window tiling comes to MacOS
Presenter View for video calling
Password app across Mac, iPhone, iPad, VisionPro, and Windows through iCloud app
iOS 18
Spaces on Home Screen and more customization
Hide apps and lock apps
Photos app gets some smarts to filter by type, create more collections, and reorder and pin them
On-device mail categorization (Primary, Transactions, Updates, Promotions)
More text effects in messages, new tapbacks with emojis
iPadOS 18
Calculator!
And Math notes in Notes
WatchOS 11
Some new training modes and health features. The Vitals app can track what your vital stats look like, compare them to your average, and what they should look like. Also, improvements to Cycle Tracking and Pregnancy Tracking.
VisionOS 2
A couple of new gestures: Palm Up and tap fingers for Home and palm down for control center. Canon has a lens that can shoot Apple Immersive Video. And 8 new countries are getting the Apple Vision Pro.
Audio & Home
You can now nod or shake your head to answer a Siri question. And TVOS got an Amazon X-ray style feature called InSight, improved enhanced dialogue, and 21:9 projector support.
FULL NOTES
VisionOS 2
Photos: spatialize existing photos with advanced machine learning
SharePlay
Palm up tap for Home
Palm down for control center
Higher display resolution and size coming later this year
Train support coming to travel mode
New APIs and frameworks to encourage developers
Canon EOS R7 getting a spatial video lens
FCP can edit spatial videos and share on Vimeo app (coming in fall)
Apple Immersive Video: the 180-degree 8K format gets new partners. BlackMagic Design made a new workflow with their cameras, DaVinci Resolve, and Apple Compressor.
Red Bull doing an Immersive Video series, also The Weeknd and Submerged short film
8 new countries, including China, Singapore, and Japan on June 28
iOS 18
Spaces on Home Screen. Dark mode tint color.
Control center lets you swipe to more pages of settings. Small icons on right.
Controls gallery lets you customize.
Also available from the Lock Screen to replace camera and flashlight (and action button)
Devs can add controls to control center (Controls API)
Privacy:
Lock an app: requires Face ID (Touch or passcode) to open app
Hide apps in a hidden apps folder that is locked
Allow apps to have access to select contacts
Devs can pair an app to accessories to just one accessory, not all Bluetooth devices
Messages:
Tapbacks add any emoji as an option
Schedule when to send
Bold, italic, underline, strikethrough messages
Text Effects: emphasize, animated emoji jump, blow away, nod, ripple, shake, etc.
Messages via satellite: iPhone 14 and later. E2E encrypted as well as SMS
Mail:
On-device categorization (Primary, Transactions, Updates, Promotions)
Digest email
Maps:
Topo maps
Parks available for offline trail
Wallet:
Tap to cash to send mail to two phones in proximity, online pay and event ticket (installments and rewards in wallet)
Journal:
???
Gaming:
Game mode: max frame rates and minimized background activity. Increased responsiveness
Photos:
Single grid library at bottom
Filter by type: Filter out screenshots
Collections: time, people and pets, memories, trips, and more. Recent Days filters out recipes and screenshots
People can do groups of people like couples or families
Pin collections to the top, order collections
Carousel of best content: Favorites, Featured (also customizable)
Audio & Home
AirPods:
Nod yes or shake no to interact with Siri
Voice isolation
Personalized Spatial Audio API for games. Need For Speed Mobile will be one of the first titles with it.
TVOS:
InSight (like Amazon Xray): swipe down on the remote for actors and song tracks
Enhanced dialogue with support for TV speakers and AirPods. Improved algorithm
Subtitles when you mute or skip back
21:9 projector support
Portraits category added to screen savers, TV and Movie moments, Snoopy screen saver
WatchOS 11
Training Mode: measure intensity and duration of workout and tell if you're above or below average
Customize summary tab in fitness app
Can adjust goals for your rings by the day of the week. And you can finally pause ring tracking for a while to keep your streaks
Vitals app: How metrics relate to the typical range, highlights those outside the range
Cycle tracking and pregnancy tracking: gestational age and modify high heart rate notifications
Smart stacks: adds widgets when it thinks you need them. Like weather at the top or translate when traveling. Devs can add to smart stacks
New translate app on Apple Watch
Check-in added to workouts to let friends track you
Gestures to launch things like tap fingers to start baby tracker
Suggests best photos for your watch face and frames it with the time
iPadOS 18
Personalize Home Screen, customize home screen DND photos app additions
Floating tab bar to help navigate to other parts of an app like Apple TV. Morphs into the sidebar when necessary - can customize (like pages, keynote, etc.) - APIs as well
SharePlay: tap and draw on the screen when screen sharing with family. Permission to remotely control iPad or iPhone
Freeform: scenes to select sections of a board
Calculator: coming to iPad, supports Apple Pencil, history, and unit conversions
Math Notes: You can just write equations using the pencil, and when you write an equals sign, the app solves it for you. Get a sum just by drawing a line under a column. Do algebra and handle variables. Can create graphs and adjust in real time
Notes: all calculator features work here too. SmartScript: improves appearance as you write thanks to on-device ML. Spell check, text wrap, copy-paste. Scratch out to erase.
Mac OS Sequoia
Tapbacks, messages, math notes, hiking topo maps all coming
Continuity: iPhone mirroring
App in dock
Mac and iPhone notifications can appear together on Mac
iPhone stays locked and in standby
Drag and drop from Mac to iPhone
Automatic window tiling
Presenter preview for video calling (FaceTime, Zoom): Can replace background
Passwords app: Mac, iPhone, iPad, iPhone VisionPro, and Windows in iCloud for Windows app
Safari:
Highlights: Uses ML to detect and highlight helpful info. Summaries and extra info without leaving the page. Reader mode includes summary and ToC
Viewer: puts video front and center with system video controls and PIP
Gaming:
Game Porting Toolkit 2 has improved Windows compatibility and improved debugging tools. Assassin's Creed Shadows coming to iPad later this year. Frostpunk 2 as well
Apple Pay on third-party browsers
Apple Intelligence
Powerful, intuitive, integrated, personal, private.
Puts generative models at the core of iPhone, iPad, and Mac in next OS
Capabilities
Language: iPhone can now prioritize notifications. Writing tools can write, summarize, and condense text.
"Can create totally original images" using photos in your library.
Images: sketch, illustration, animation (built into apps across the system)
ACTION: Show me all the photos of Eileen, Sawyer, and me. Play the podcast my wife sent me the other day. Can reference content on screen
PERSONAL CONTEXT: Knows when meetings are, where you are, etc. Draws from mail, maps, etc.
Architecture (Privacy)
On-device processing
Large Language and Diffusion models
On-device semantic index from across apps
Private Cloud Compute:
Servers made with Apple Silicon. Sends only data relevant to the task to servers only when necessary. Data is never stored or exposed to Apple.
Code can be inspected by independent experts
Private cloud compute cryptographically locks cloud to your phone, won't run unless software can be inspected.
Experience
Siri:
Glowing light wraps around the edge of the screen
Recognizes more natural language including stumbles, understands context from previous requests
Can also type to Siri
Can ask questions about how your devices work for help content
On-screen awareness: "Add this address to context card"
Can take in-app actions
App Intents API: Devs can let Siri take actions in their apps
Siri has awareness of personal context, like photos, messages, calendar events, and files.
Can find a picture of your DL, get ID number, and enter it into a web form.
Rewrite:
Can rewrite things like emails, offering multiple versions. Choose tones: friendly, pro, concise
Make it read like a poem
Proofread: review edits individually or accept all
Summarize: can reduce long emails to add as TLDR
Can access system-wide including third-party apps
Smart Reply in mail: suggestions for responses based on email. Asks questions to customize response
Mail can show summaries in email list
Put priority emails at the top
Notifications:
Priority notifications at the top and summarize them
Reduce interruptions to only surface those that need immediate attention
Genmoji:
Describe the emoji you want to generate it. Create a Genmoji from a person in photo library
Image Playground: in things like messages or in-app - image generator that works on-device. Animation, illustration, or sketch. Image Playground API
New tool in the Notes app called Image Wand that takes a rough sketch and turns it into a better image
Clean Up for photos
Natural language for photos search
Use natural language to create a video clip
New AI recording and transcriptions in Notes — and the Phone app!
ChatGPT 4.0: coming to Siri and Rewrite - asks if you want to use ChatGPT. Free. No account. Won't be logged. Can connect account to access paid features.
OTHER AI MODELS COMING IN FUTURE: if they can reach deals. Apple Cash at work.
"AI for the rest of us"
Apple Intelligence coming to the iPhone 15 Pro and iPad and Macs with M1 and later. All with the new OSes, of course.