Accessing Media Files (Audio, Video, Images…) From DigitalOcean Spaces Using Java

Recently, I need to upgrade my English learning app on Android to use multiple servers. Previously, I used two servers already. However, the way I manage media files caused me a lot of headache. So, I decided to switch to DigitalOcean since they have servers around the world and the pricing is good ($5 for 250GB storage and 1TB monthly transfer, which is good).

The initial problem with DigitalOcean spaces

Coming from old-school servers, I thought I can upload the files to DigitalOcean spaces using FTP and access the files directly as I usually used. However, the first problem is you cannot upload to spaces using FTP and the second problem is in order to make a file accessible to public, you need to set it permission to public. If you have one or two files, that wouldn’t be a problem.

I have more than 18,000 files.

The solutions to uploading multiple files

It turned out that DigitalOcean spaces use Amazon s3 technologies. So, in order to upload to spaces, you need to have FileZilla Pro (paid) or CyberDuck(free). I chose FileZilla Pro because it allows you to resume failed transfers while CyberDuck doesn’t allow that (or I couldn’t find where is that option).

The solution to accessing files

As my app used by thousands of people, the media files need to be accessible to all of the users. Actually, it is very easy to generate a public accessible URL for any files (they are called objects) using the aws-android-sdk. In case you need a working sample code, here it is:

public class DigitalOcean implements MediaServer{

    private static final String KEY = "YOUR_KEY";
    private static final String SECRETS = "YOUR_SECRETS";

    private String endpoint, bucketName;
    private AmazonS3Client s3Client;


    @Override
    public String getEndpoint() {
        return endpoint;
    }

    @Override
    public String getType() {
        return "s3";
    }

    public DigitalOcean(String endpoint, String bucketName)
    {
        this.bucketName = bucketName;
        this.endpoint = endpoint;
        AWSCredentials myCredentials = new BasicAWSCredentials(
                KEY,
                SECRETS
        );

        s3Client = new AmazonS3Client(myCredentials);

        s3Client.setEndpoint(this.endpoint);
    }

    public String getMediaURL(String objectPath)
    {
        GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(bucketName, objectPath);
        URL objectURL = s3Client.generatePresignedUrl(request);

        return objectURL.toString();
    }




}

As you can see, I created an object to generate URL to any files(objects). For example, here is a sample space url:

The bucket name would be data-sample.

If your file, let call it song.mp3 located at /data-sample/media/song.mp3, the objectPath would be: /media/song.mp3.

If you run the method getMediaURL(), you will get the path to the file that is accessible to your users but you don’t have to mark the file public.

https://data-sample.sgp1.digitaloceanspaces.com/media/song.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180925T060309Z&X-Amz-SignedHeaders=host&X-Amz-Expires=899&X-Amz-Credential=BP5K25LRRQWYV2CMHZLL%2F20180925%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=6a21133ced60047cfba60053afcdf712997c474585ade2f0050bc853b0a4db89

 

How To Combine Multiple Text Files (.txt) Into One Big File

Recently, I have a specific need to combine multiple files into one big file. The task is very simple. However, if I do it manually (combine over 32,000 small text files into one), it may cost me a day, or even more.

How to combine multiple text files into one

So, I decided to make a tool in Java. Its interface is very simple but you can use it right away without much learning. Here is the app’s interface:

As you can see from the interface, there are three buttons and these are the one you need to pay attention to. To start combining  your small .txt files, click on the button “Select files to combine”. Here, I’m going to select all of my .txt file (over 32,000 files). For those of you who are curious, these are the books from project Gutenberg.

Now, click on Open.

Next, you need to select the destination file. This is the file that contains all the text from the small files.

You can see that the path to the destination file is displayed in the text field at the left of the select button.

Now, simply click on combine now. The program will go ahead and combine your files into one file. In my case, all over 32,000 files are combined into one and the destination file is over 250MB! That’s a lot for a .txt file. In fact, there are more than 7 million line of text inside that file.

If you are a Java developer,  you can check out the repo here on Github here to modify it to your needs. If you are an user, you can download the ready built app here to start combining your .txt files:

Click here to download the executable app.

How To Filter Non Alphanumerics Characters From Your String And Its Application In Excel

If you are like me, I have to deal with text a lot in my work. As a programmer, I need to find and match string. Recently, a friend of mine, who is an accountant gave me a list of books’ titles. Most of them are fine. However, there are some contains strange characters and she wanted to remove such titles.

As you can see in the picture, there titles with @ and Chinese characters. Using filters doesn’t work because there are more than 1000 rows. That was when I said to myself: “I need a tool”.

So, I made one for this specific purpose: “to help my friend remove all the strange characters in her Excel workbook”.

Here is how the tool look like:

As you can see, you can put the original text in the source string box, adjust a few options above and click on filter. The result will be shown in the Result string box:

As you can see that all the rows that contain strange characters are removed.

However, what if you have some special characters that is allowed to appear in the title? No worries! You can put that into the Allowed characters box and separated them by spaces.

For example, It is OK for me to have the character @ in the title, I would put @ into the Allowed characters box. In addition, that title Blog@books.de contains the character dot (.) so you need to put that into the list of allowed characters too.

Let’s see what we’ll have:

As you can see now, the title with . and @ is accepted.

There are also two checkboxes that allowed you to filter the text further. You can decide to allow numbers or spaces in the title by checking these checkboxes.

I put the full source code for the tool here:

https://github.com/datmt/Remove-Lines-Contain-NonAlphaNumeric-Characters

If you are not a programmer, you can download the application here to start using it:

Download the .jar file

If you have any suggestions, please let me know. Thank you very much!

Extract List Of Unique Words From a Book, Article (Supports .txt, .pdf format)

As an ESL, I usually wonder: “how many words I need to know to speak English well?”. I read somewhere that you don’t need more than 2,000 words to carry out 99% of daily conversations. 2,000 words don’t seem to be a lot and if I can learn 10 words a day, I can be conversational in any language in just 200 days (less than a year).

So, what if I want to enjoy the novels, books in English, how many words I need to know.

I doubt the number would be a lot more than 2,000 words.

So I made a tool to read all the text in a book (you can select more than one book in .pdf or .txt format) and it will export all the words in that book into a table.

Here are some results:

Adventures of Huckleberry Finn

As you can see at the top, there are 6461 different words in this book.

Wuthering Heights

for Wuthering Heights, you need to know 9,456 words.

That’s a lot of words.

The numbers aren’t 100% correct. There are some gibberish such as website address, two letter (like country codes) may get in the list. However, you can roughly estimate that there are 95% of the numbers are real words.

I find that this is a good way to quickly find out new words that I don’t know. I can click on the words to get the definition (if available) on the right.

You can download the program here for free. If you have any suggestion, please let me know.

Download Word Extraction app here