Mystery Meat 2.0 – Making hidden mobile interactions accessible

Mystery Meat 2.0

  • Ted Drake, Intuit Accessibility
  • Poonam Tathavadkar, TurboTax
  • CSUN 2017
  • Slides:

This presentation was created for the CSUN 2017 conference. It introduces several hidden interactions available within Android and iOS. Learn how these work and how to make them accessible.
Blue Bell Ice Cream Blue Bell Ice Cream web site with mystery meat navigationis a classic example still live on the web.

The user must hover over the different images to see what they represent. It uses an image map and lacks alt text.

Android Touch and Hold

A.K.A.: Android’s Right Click or Android Long Press to Add context-specific menus

  • Default: Touch and hold
  • With TalkBack:
    Double tap and hold to long press

Mint Transactions

This short video shows how you can use the touch and hold/long press to quickly change category or merchant name within the Mint application


  • onLongClick: Called when a view has been clicked
    and held
  • Define and Show your menu
  • Not for vital interactions.
  • This is a short

It is possible to modify the default notification to the user

iOS 3D Touch

iOS 3D Touch was introduced on the iPhone 6S. It detects the pressure a person applies to the screen with their finger. I light touch is seen as a tap. A medium touch will
trigger a peek view. A continued firm touch will launch the peek’s content into a full screen.

This also allows a person to trigger a shortcut menu on app icons.

  • Peek:
    Quick glance at relevant information and
  • Pop:
    Open full content previewed in the Peek
  • Quick Actions:
    Custom task list from app icon

User Experience: A light press opens a hovering window so you can “Peek” at the content. When you press just a little bit harder, you will “Pop” into the actual content you’d just been
previewing in a Peek.

Developer info

Quick Actions

This short video shows how 3d touch is also available via the app’s icon for quick tasks.

Pressing and holding the ItsDeductible icon will trigger a menu with customized tasks. App Icon Developer resources


Swipe Revealed Actions

Alternative actions allow users to quickly make changes without having to open a detail screen. For instance, they can delete a transaction or change an email’s status. The standard interface is to display the options when a user swipes a row to the left. For voiceOver users, the options are announced as alternate actions

It’s Deductible Actions

This short video shows how the alternative actions menu is used in standard mode and how VoiceOver announces the options.

In iOS, editActionsForRowAtIndexPath defines the actions to display in response to swiping the specified row

  • Accessible by default
  • Define:
    • Target Action and Response
    • Visible Text
    • Background color

Swipe Based Navigation

TurboTax uses a custom swipe based navigation between views. It lacks button or suggestions to move back and forth. User testing has showed it to be effective for
sighted users, but required some extra work for accessibility.

Default Experience With VoiceOver

The default experience on Turbo Tax uses a custom swipe gesture that lacks navigation buttons.

TurboTax detects a user’s Screen Reader/Switch Control status to show navigation buttons on Android and iOS

This video shows the default and VoiceOver/SwitchControl experience.

Notice in the standard experience how the screen tracks the user’s finger movement. This is not a standard swipe gesture, so it will not work with VoiceOver enabled.

We detect VoiceOver and SwitchControl is running to display alternate back and continue buttons

Swipe Navigation

  • Animated transition between views
  • Next and Back flow with every screen
  • Eliminates navigation buttons
  • No buttons? Accessibility?
  • Have I reached the end of the screen?

Instead of a basic swipe gesture, this interface tracks the movement of the finger across the screen. This allows the animation to match the user’s finger speed for a more
natural transition.

However, the finger touch is intercepted by VoiceOver, so the custom navigation does not work when VoiceOver is enabled.

Detect Accessibility ON

UIAccessibility kit provides two methods

True == Show Navigation Buttons

These booleans always return true or false. We use this to insert navigation buttons into the screen.

State Change Notification

This does not solve for the more complex of when the user decides to turn it on/off in the middle of the flow of the application for changes to take place dynamically.

For that as well, iOS has great support. Fires an accessibilityChanged event that helps detect changes even when the user is in the middle of the flow and chooses to
turn voice over on/off.

User enables VoiceOver while on a screen

Detect the status change

TurboTax Helper Function

  • How can we refactor code to detect any
    accessibility related settings and address them
  • Helper function to the rescue!
  • NSNotificationCenter adds observers to track any
    settings that may require us to show buttons.
  • This is an OR logic. Example – if voice over OR
    switch control status changed, display buttons.

Code specifics

  • Boolean is assigned a value – true if buttons need to be shown.
  • Consider a React Native project, all this happens in the native code side (Objective C). This boolean is then handed over to the JAVASCRIPT side since it is not feasible
    for Javascript to get information directly from the device.

The low-vision experience with an iPad

I’ve met many small business owners with low vision that love using an iPad, or other tablet, as their key device. The iPad allows entrepreneurs to use a single device to handle all of their business tasks, such as point of sale, payroll, accounting, purchasing, and creating estimates on location. As a person with low vision, the built-in zooming functionality makes it easy to explore the screen and interact with the elements.

Karo Caran introduces how a person with low vision uses an iPad in this video

She recently created this new vidoe for a presentation on design for low-vision users. There are some key takeaways, such as consistency of banners and using background colors to differentiate between header, body, and footer.

Inheriting user preferences

While zooming is a great way to review content, not all low vision users want to use their applications in this manner. Many simply want to user the built in settings for higher contrast, bigger text, and reduced transparency to use their iPad without continually zooming in and out.

Your applications can automatically inherit these functions if you use standard iOS/Android components and the iOS Dynamic Type. If the app isn’t using Dynamic Type, we’ll need to check for a user’s display preferences and modify the fonts, colors, animation, etc.

Bold Text

The following code checks to see if a person has bold text enabled in their preferences. If so, use Avenir medium instead of light

+ (NSString *)fontName {
    if (UIAccessibilityIsBoldTextEnabled()) {
            return @"Avenir-Medium";
    return @"Avenir-Light";

Darker Colors

We can detect if a user prefers darker colors, this is especially helpful for links that sit on colored backgrounds

+ (UIColor *)detailColor {
    if (UIAccessibilityDarkerSystemColorsEnabled()) {
        return [UIColor blackColor];
    return [UIColor grayColor];

Additional Checks

Block level links and screen reader support

Daniel Göransson introduced VoiceOver’s contradictory support for block level links on a mailing list for Apple accessibility. Specifically, Safari + VoiceOver on OSX allows the link to be announced as a single object, whereas on iOS, the link is announced as individual elements.

Prior to HTML5, the link tag was an inline element and was not meant to contain block level elements, such as headers. This led engineers to create multiple links as designs included teasers and id card modules. These may include a thumbnail photo, header, abstract, and read more link. Bruce Lawson describes this code simplicity: Block level links in HTML5.

Göransson describes the desktop vs. phone behavior:

In Mac Safari and VoiceOver using tab key all items get announced and works perfectly. But in iOS Safari and VO using swipe to next item or touch exploring, each item get announced separately.

This is actually the intended behavior, as Chris Fleizach explains:

iOS is much flatter than MacOS, so we want to give individual access to each element within a link. Whereas, on the Mac, you can navigate inside of that link group if you so desire.

So how could we trigger mobile users to hear the content as a single chunk instead of individual elements? The following code snippets are inspired by Göransson’s emails. He deserves credit for the ideas, I can accept blame for mis-representation.

Please Note: you may find it easier to view these examples outside the WordPress framework. Visit the block level example page.

Base code snippet

This example has a header, paragraph, and image. I’ve placed alt=”” on the image, as the surrounding text provides adequate description.

<a href="">
  <h4>The Deësis mosaic</h4>
    <img src="..." alt="">
    The Deësis mosaic ... </p>
  <p class="more">See the larger image on Flickr<p>

The Deësis mosaic

The Deësis mosaic (Δέησις, “Entreaty”) probably dates from 1261. It was commissioned to mark the end of 57 years of Roman Catholic use and the return to the Orthodox faith. It is the third panel situated in the imperial enclosure of the upper galleries. It is widely considered the finest in Hagia Sophia, because of the softness of the features, the humane expressions and the tones of the mosaic.

See the larger image on Flickr

Using ARIA to make the experience consistent

ARIA roles allow developers to define the purpose of content. Ideally, you should use the correct tag before even considering ARIA attributes. Use the role when HTML doesn’t provide an adequate tag (tabs), or the HTML equivalent is not as well supported by accessibility APIs (nav). Secondarily, roles are used to restore the semantics of code when the correct tag is not used (div based buttons).

This example is correctly using the link tag, as the user will be taken to a new source of content, such as a web site. I had initially suggested using role=”button”, as that would flatten the experience and should announce the entire content, regardless of platform.

<a href="" role="button">
  <h4>The Deësis mosaic</h4>
    <img src="..." alt="">
    The Deësis mosaic ... </p>
  <p class="more">See the larger image on Flickr<p>

The Deësis mosaic

The Deësis mosaic …

Using role=”button.

This would change the user’s expectation, as it announces this as a button instead of a link. It implies clicking on the object would trigger an action. But what is the action? Buttons should have a clear and concise call to action (“print”, “share”, “close”). This also means the link would appear within the quick menu for forms, rather than links.

Combining with aria-hidden

Göransson also noticed an unusual problem when placing aria-hidden=”true” on the image. This removed the image’s area from the touchable region.

Also when using anything with aria-hidden=”true” inside that <a>, the area where hidden item is located will become untouchable even if it’s still inside a <a>. For example, if I float an icon/image with aria-hidden to the left — that part of the <a> will be subtracted from the hitbox.

<a href="" role="button">
  <h4>The Deësis mosaic</h4>
    <img src="..." aria-hidden="true" alt="">
    The Deësis mosaic ... </p>
  <p class="more">See the larger image on Flickr<p>

The Deësis mosaic

The Deësis mosaic …

Added aria-hidden=”true” to the image.

Victor Tsaran re-iterated Chris Fleizach’s suggestion of using role=”text” to allow the block to be announced as a single unit.

I am not sure you can achieve this in Mobile Safari without perhaps having to resort to the role=text. The Voice Over navigation of the DOM is much flatter on iOS than on Mac OS and you are inserting a bunch of block elements into an inline one.

<a href="" role="text">
  <h4>The Deësis mosaic</h4>
    <img src="..." aria-hidden="true" alt="">
    The Deësis mosaic ... </p>
  <p class="more">See the larger image on Flickr<p>

The Deësis mosaic

The Deësis mosaic …

Using role=”text” on the link

But this changes the semantics of the link to a text node, removing it from the link menu.

Göransson’s next attempt wraps the link’s content in a div with role=”text”. This keeps the link’s semantics and the entire content is announced. However, this also removes the semantics of the h4, removing it from the header menu.

<a href="">
  <div role="text">
    <h4>The Deësis mosaic</h4>
      <img src="..." aria-hidden="true" alt="">
      The Deësis mosaic ... </p>
    <p class="more">See the larger image on Flickr<p>

The Deësis mosaic

The Deësis mosaic …

Wrapping the content within a div that has role=”text”

What is the answer?

This exploration could be summed up by Sina Barham’s tweet about trying to change the expected gestures.

While we could solve the issue of making a block level link work consistently across mobile and desktop browsers, that may not be the ideal solution. The user’s expectations are context dependent, this is especially true with navigation.

Accessibility Links: iOS Developer Resources

Periodically I go through my bookmarks to organize and make sure they are still relevant. There’s no reason to keep them bottled up in my browser’s toolbar, so I’m sharing them via the Accessibility Links category.

I use the following links regularly while doing iOS evaluations and research.Continue Reading Accessibility Links: iOS Developer Resources

Developing a mobile accessibility strategy

This presentation was created for the TechShare conference in Delhi, India Feb. 2014 It shows how Intuit’s mobile strategy has encouraged accessible mobile applications. The secret behind Intuit’s mobile accessibility strategy is that it has less to do with accessibility and everything to do with user experience and user-based design.

Continue Reading Developing a mobile accessibility strategy