SKYLER ARABAK

FrameSlice.com

• Client side video cropping and labelling site
• Uses ffmpeg.wasm to avoid server uploads
• Next.js / React / Material UI

iCodeThis.com/SkylerA

While I wouldn't claim artistic vision is my forte (as evidenced by this site 😉), I am skilled in slicing other people's designs. You can check some of my work on iCodeThis where I recreate designs from only a png of the desired design.

Video Processing Pipeline

In my freetime, I'm working on a python framework that seeks to simplify the creation and use of computer vision models for domain specific projects as well as exploring how to make computer vision accessible to the common user.

• Python Back-end
• • Crop and Pre-process video frames
  • Apply user computer vision models to frames
  • Collect and interpret results
  • Serve results via Flask
• React Front-end
• • Collect frames from video for feedback/testing
  • Create and modify crop and filter tasks
  • Test and debug user model results
  • Also experimenting with pulling backend elements like video processing into the client app
  • • Client side video parsing proof of concept
    • Early testing with tensorflow.js indicates promising results for in browser inference for smaller models. (Demo below in features)

Making Filters

"Filters" are a collection of actions run on a given frame of video. A filter can take a whole video frame or can crop down to a specific location. A filter can apply custom pre processing functions to its crop. The augmented crop is then passed to a specified parsing/inference function which allows the user/dev to run whatever inference or other data collection they want on that crop. The results are stored per crop, per frame and are returned to the front end for debugging purposes.

In this example, the user quickly selectcs two portions of the video they'd like to crop for OCR later.

Crop Image Pre-Processing

Pre-procs allow the user to modify their image/data before it is passed to an inference/parsing stage.

On the left 2 filters are setup to apply different image pre processing steps to improve the images for OCR. On the right, those filters are run for the next 30 frames and the results are shown.

Filter Results

Developers can return whatever data they want in their results.

In this example, a mobilnetv3 classifier is run on 4 different crops and inference info is returned with it.

Filter Timeline

Provides a visual timeline of classifications made in a given range of frames.

The timeline is an interactive canvas using svgs that are generated on demand using class/marker meta data and then cached for later drawings. Clicking a result jumps to the frame it was captured in to allow for quick debugging/verification of inference results.

Label Editor

Client side label editor that can load, sort, and edit classification labels. Greatly simplifies finding incorrectly labeled classes as all labels are shown together making different images stick out. Labels can be downloaded as a zip or uploaded to a given model training label folder.

In this example, the class editor is pointed at a local collection of files. When some mislabeled classes are found, they are quickly fixed and then a few images are given a new label for demonstration purposes.

Annotating Results

A filter's results can be selected and annotated to capture new training data or to capture and re-label images the model is currently mis-classifying

Client Side Video Processing

This is the first step in moving video parsing to the client side. The goal is to leverage existing user hardware to remove video uploads and reduce server load. My vision is to provide a free tier that allows a user to leverage their hardware and then provide server powered instances as an upgrade. Live Demo

Client Side Video Inference/Classification

Using tensorflow.js to classify crops taken with ffmpeg.wasm and display them in a custom timeline component that syncs up with the loaded video.

Experimental Concept: Time Based Labelling

In this example, a range of similar, time-sorted frames are overlayed on top of each other and the user quickly labels each portion. More Explanation and Live Demo