Date
2022/12
Role
Product Designer
Responsibility
UI/UX Design
Co-workers
Product Manager, Marketing Analyst, iOS Engineers
01. Objective
By developing a MVP that can be showcased on Dynamic Island, we aim to not only gain more promotion opportunities but also better understand app user behavior.
02. Role & Deliverables
I was responsible for MVP design, including user flows and mockups, and actively participated in A/B testing discussions to ensure effective assumption verification through design.
03. Challenges
At the time when Dynamic Island was just announced, everyone was not very familiar with its design principles and development feasibility. This led to more time spent on communication and exploring the possibilities.
04. Outcome & Impact
The test result was insignificant, but we still observed some app users' behavior and opportunities.
Project Background
Hahow is an online learning platform that began by offering fundraising video courses. Its aim is to provide a platform where anyone can learn any skill at any time and place. We also launched iOS app in 2021 and continue to operate it.
After the announcement of Dynamic Island, Apple Greater China launched the 'Challenge on the Island: Adapting Live Events and Dynamic Island' online campaign, inviting development teams to adapt Live Events and Dynamic Island by December 20, 2022. Successful teams will gain enhanced app promotion opportunities.
Considering we had enough resources and time, and also an opportunity to collect more behavior data from app users, we decided to join the campaign.
Brainstorming
We conducted a brainstorming meeting to determine which features would be most suitable for us. The key considerations included:
- Feasibility as an MVP
- Enhanced User Data Collection
- Alignment with Dynamic Island's Primary Goals
The feature should be quickly developable as an MVP (Minimum Viable Product).
It should allow us to gather more behavioral data from our app users.
We also considered how the feature aligns with Dynamic Island's main objectives, which include tracking tasks and events and providing persistent locations for frequently updated information.
Conclusion
Aligning with our product's unique features, we chose to introduce a 'Learning Records' feature in conjunction with Dynamic Island, while simultaneously conducting A/B Testing to collect user behavior data.
🖌 What is Learning Records?By recording the time users spend learning on the app, it informs them whether they have reached the daily learning goals.
📌 What are the Rules?Users who consistently achieve 15 minutes of learning per day for at least 5 days a week will be rewarded with a learning medal. This test would run for 4 weeks.
Assumption
A/B Testing
Without Learning Records
With Learning Records
📍 Testing Goal
- To verify the impact of the Learning Records function on user learning behavior, retention rates, and the frequency of app usage.
- To compare the percentage of users achieving 15 minutes of daily learning time in both the Control Group and Treatment Group.
- To cultivate the habit of using 15-minute time fragments for learning among users.
Mockup Design
01. Learning Records
We've added a Learning Records section to the Home page, tracking users' daily learning time.
The section's status changes based on the progress users make over time.
/w=3840,quality=90,fit=scale-down)
02. Learning Records on Dynamic Island
When users are watching courses in the app, devices equipped with Dynamic Island can display Learning Records.
This feature enables users to easily track their learning progress, check whether they've met their goals, and stay motivated to achieve them.
03. States of Learning Records
We've created various learning states to match every stage of the learning process, including 'not started,' 'in progress,' 'daily goal achieved,' and 'earning the learning medal.' This allows users to easily identify their current status at a glance.
04. Earning Learning Medals
When users achieve 15 minutes of learning per day for at least 5 days a week, they receive a learning medal that they can share on social media.
We've designed four different learning medals, allowing users to earn a new one each week and motivating them to keep learning consistently.
4 kinds of learning medals.
The Learning Records section on the Home page also displays the progress of medal collection.
Outcome & Findings
⚠️ Only capture parts of the findings to share.
A/B Testing results : Insignificant
💡 Finding 01
The rules of Learning Records themselves didn't encourage users to start learning but rather motivate those who already have learning habits to add an extra 15 minutes to achieve daily goal.
💡 Finding 02
Achieving the daily goal of 15 minutes was too easy, but attaining it for 5 days a week was too difficult. Possible reasons include:
- Inappropriate Rules
- Lack of Motivation
Initially, our app lacked learning behavior data, so we established rules based solely on team consensus without a data foundation, which may have affected the results.
The treatment group averaged 40 minutes of learning per week. However, when they attempted to increase their learning time in the third week, they didn't experience any significant changes or receive the desired rewards, leading to a decrease in activity in the fourth week.
Takeaways
01. Gathering User Learning Behavior Data Before A/B Testing
From this experience, we've learned that relying solely on team consensus for testing rules carries a high level of risk. It's more efficient to collect relevant data before initiating any testing.
02. Valuable Data Gained Despite Insignificant Testing Results
While we didn't discover an effective method to encourage users to extend their learning time, we obtained valuable data from this experiment that will provide a stronger foundation for our next experiment.
Want more? Check out my other projects below.
![[Design Ops] Maintenance Rule Iteration](https://images.spr.so/cdn-cgi/imagedelivery/j42No7y-dcokJuNgXeA0ig/cf0d316c-e97f-4fa4-b1c9-cee53c1b6ff8/Untitled/w=1920,quality=90,fit=scale-down)
