Reflections from a Product Management Apprentice

Originally posted on NYC Opportunity’s Medium Channel.

I’ve just finished a five-month internship with the Mayor’s Office for Economic Opportunity as Product Management Apprentice. I am grateful for the opportunity to have learned from and worked with this thoughtful and dedicated team of technologists and builders. I hope that sharing my experiences will not only showcase this team’s great work, but also spur further attention and analysis towards the need for improving products and services in the public sector.

When applying, I was primarily drawn to NYC Opportunity’s mission: to reduce poverty and increase equity, advancing the use of research, data, and design in the City’s program and policy development, service delivery, and budget decisions. I can not think of a more noble cause to which technology should be applied, nor a better place to learn. Working at NYC Opportunity also gave me the opportunity to better understand the product management discipline and life in the public sector while finishing my MBA at Cornell Tech, a new product-oriented graduate program located on Roosevelt Island.

As a Product Management Apprentice, I mostly worked on ACCESS NYC, a web application for the public which helps benefits seekers and navigators identify, screen for, and prepare to enroll in government benefits. In the beginning, my weekly schedule merged education with practice, and looked kind of like this:

Tuesday class: Social Entrepreneurship at Cornell Tech

NYC Opportunity: experiment with using the UN’s Sustainable Development goals to identify proxy metrics to help measure offline success for ACCESS NYC.

Wednesday class: Leading Digital Innovation and Transformation

NYC Opportunity: Help research new digital identity initiatives for the team’s next product, better understand the problem that should be solved, and learn from similar products’ successes and failures.

Thursday class: Product Management

NYC Opportunity: Sit in on sprints and retrospectives and observe how the product team adapts those practices to the unique needs of a government product.

(My schedule did not line up so perfectly in real life, but it was close!)

Once I graduated from school, I was able to dive headfirst into setting up Google Tag Manager, Google Optimize and Data Studio for the ACCESS NYC team to help them better understand user’s behavior and flow through the site. This involved learning RegEx, sharpening my HTML, SQL and coding skills, and working with the more technically savvy fellows and developers in the office to ensure we were implementing the right scripts and formatting our tables correctly.

My personal goals–to develop technical fluency and to better understand product development and execution–were certainly achieved to the extent I had hoped for. I have not only learned about the broad role and many challenges of a Product Manager, but also met lovely and interesting people and made new friends. My biggest thanks go to Song Hia, Hilina Mohammed, Devon Hirth and Steven Aguilar who demonstrated ample patience and mentorship throughout my many overlapping learning curves.

Image for post
The Learning Curve
Image for post
What it felt like
Image for post
Also what it felt like
Image for post
What it probably looked like

Because of this apprenticeship. I’ve also come away with bigger questions that have reinforced my desire to remain working alongside the public sector for the time being. In classic PM “how might we” phrasing, below are three questions I hope to explore further:

How might we improve a product to increase benefits access for every citizen, especially those beneath the poverty line?

User journeys are critical pieces of every product’s development and success. How will the user flow through a site and achieve their goal? What is their goal, and how do we determine this? How can we make our site more accessible and for whom?

My interpretation of a user journey is heavily influenced by marketing funnels, loops and driving revenue-generation. Therefore ACCESS NYC, a non-revenue-generating web screener, proved to be an interesting challenge. From my experience, the better you are able to frame and articulate the details of each user journey, the easier it is to identify areas for improvement. How do you build user journeys when the product serves millions of New Yorkers near, at, or under the poverty line?

For ACCESS NYC the two main users are benefits seekers (residents seeking government benefits) and benefits navigators (case workers or others who assist benefits seekers). The website is translated into 11 languages and content is presented in a digestible way for every level of literacy. The challenge of further segmentation into archetypes depends on how much data you want to collect, how much time and capital you have to devote, and whether the results will move the needle in a meaningful way. All three of these challenges have additional trade-offs in government product development.

Additionally, the word “access” has been discussed in various capacities on our product team, whether it means adhering to web accessibility standards or designing an experience oriented around “constraint thinking” that often affects those living in poverty. Design tools and tactics play a large part in addressing and building solutions around these concepts. I love that NYC Opportunity uniquely has a dedicated Service Design Team who creates and employs frameworks to ensure better service delivery and product experience.

How might a government measure digital product impact and success?

One of my main responsibilities as an apprentice was defining actionable metrics that could help the product team optimize ACCESS NYC’s design and user experience. This involved first mapping out all relevant metrics along a typical benefit seeker or navigator’s journey then determining if or how we could collect that data.

Some tougher questions arose:

  • How might NYC Opportunity measure whether a user signed up for an agency benefit via ACCESS NYC when the sign up process happens offline and the agency does not track or report on referrals?
  • If there is a government shutdown and more people want to enroll in government benefits, ACCESS’s web visits increase. How do we improve the site’s usability while keeping external factors such as political, societal, and seasonal factors constant? (One answer: A/B testing!)
  • One of ACCESS’s major goals is to direct traffic to external benefit providers. In other words, away from the site. Therefore traditionally prioritized metrics such as bounce rate or pages per session are not as important as exits or referrals. How do we know users are finding and getting the most out of the content and resources ACCESS offers?
  • What is the best way to share success with stakeholders or government agency leaders who may not measure success in terms of revenue, data or graphs? Should government product teams dive deeper into tools and frameworks offered by mission-driven companies, non-profits or agencies such as B-Corp, impact investors, or the UN?

One approach to optimizing the site and reducing external factors is in the form of A/B testing. I was happy to help set up and launch the first round of A/B tests for ACCESS NYC, with the goal of helping to build the process into their product development processes down the road.

How might the government improve communication and collaboration across silos?

I’ve always loved implementing a good process, though understand that outdated processes are often the very problem with government in the first place. The ability to adapt or change quickly in a highly structured environment is difficult, thus why agile product teams often are small and cross-functional. NYCO has done a good job of creating agile teams within a government context: for example, ACCESS shares monthly metric reports across teams and with stakeholders and sprints are cycled through every few weeks. However, when working within a system with ingrained norms and systems, it is a constant challenge to overcome.

This was the topic of many all-staff meetings, with the understanding that building communication channels between teams (and even between teams within teams) can be an additional challenge.

Image for post

Often, the hardest problems are the most important to solve. I admire NYC Opportunity’s steadfast dedication to addressing questions like these, taking one step at a time to help alleviate poverty and improve NYC residents’ access to benefits and social services. Thank you to the whole NYC Opportunity team for your kindness, service, dedication, your mentorship, and I look forward to continuing to build upon everything I’ve learned here.

This post is a part of the Service Design Studio at the NYC Mayor’s Office for Economic Opportunity. The Studio works to make city services more accessible and effective for low-income New Yorkers. To learn more about our work, visit our website.

Image for post

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: