E3x2D4gT

1Inspiration
21 in 5 children in the United States have learning disabilities and attention disorders like dyslexia, ADHD, Dyscalculia, Dysgraphia, Auditory Processing Disorder (APD) and Language Processing Disorder (LPD). Individuals with Dyslexia face significant challenges in educational settings, team presentations and meetings where a lot of the presentation is solely text, numbers and/or graphs. (https://ldaamerica.org/types-of-learning-disabilities/)
3
4Some of the specific challenges faced by individuals lying on the spectrum of learning disabilities face are:
5
6Difficulty reading normal text (Dyslexia)
7context blindness (figuring out senses of different words in different scenarios) and
8Trouble focusing on dense, solely textual material without visualizations.
9This is where readAR comes in. Through an array of services built into AR, our project aims at enhancing the learning and perceptory experience for individuals with these conditions.
10
11What it does
12readAR's main purpose is to make it easier for these individuals to learn, whether that be about what "work" actually refers to in a sentence, or understanding dense physics slides. We do this through using AR to re-render the world in a more dyslexic friendly font (like Dyslexie), and giving the option to users to add machine-learning inferred context to key phrases in the sentence. This enables us to pull visualizations and images that engage the user and enable him/her to better understand the concept. We also do SpeechtoText to transcribe the entire lecture/ meeting (especially helps individuals with auditory processing disabilities) for future reference. This also enables us to do intelligent quiz generation to assess the user's weak points in his understanding of a specific topic, and then give him customized resource recommendations. (Pulled using Bing's Custom Search API)
13
14How we built it
15We built the AR components and mobile app using Swift and ARKit. We also managed to to re-implement a pretty neat paper dealing with word disambiguation (Wiedemann, G., Remus, S., Chawla, A., Biemann, C. 2019) and serve it on the cloud (this is what allows us to provide context-specific definitions!). We utilized Azure's neat APIs, which allowed us to provide near-realtime OCR, intelligent search for resources and image suggestions, To top it all off, (almost) everything was Dockerized and deployed through Oracle Cloud Container Service to allow for some pretty nice scaleability.
16
17Challenges we ran into
18We all went into the hackathon knowing that this idea would be crazy -- a moonshot even. And that's what made it so challenging and fun. Some of the most significant challenges we faced were,
19
20Finding a good way to render all the additional information without being overwhelming for the user
21Training and serving our own word disambiguation model that was this accurate (there were almost zero resources about this)
22Figuring out how to use AR for the first time
23Accomplishments that we're proud of
24One problem that we noticed in existing systems was how poorly they handled WSD (Word-sense disambiguation). We recognized this as an important issue for words in specific fields that have domain-specific meanings. E.g. "work" in the formal physics sense vs the general meaning. This is evident in the examples below:
25
26image
27
28image
29
30We tackled this by creating and fine-tuning our own BERT model to be able to accurately disambiguate word meaning, through which we were able to reach 77% accuracy (5% off from the state of the art) in around 1.5 hours on our laptops.
31
32What we learned
33We learned to use and integrate a pretty large array of different services- ranging from our own ML models for WSD and Question Generation, Azure Cognitive Services, Oracle for deployment, with Swift for our AR app. It was quite the journey putting all these jigsaw pieces together.
34
35What's next for readAR
36We believe we've stumbled across an idea that definitely has significant potential to be taken forward and make an tangible impact. We hope to continue refining the project and hopefully also have it beta-tested by actual users with these disabilities.