Handling Files

Skapi database is integrated with Skapi's cloud storage and CDN. This allows you to upload any size of binary files to the database without any additional setup.

Uploading Files

To upload files, you can pass the HTML form SubmitEvent or FormData that includes FileList object when calling the postRecord() method.

Additionally, We can log the progress of the upload by passing a ProgressCallback in the progress parameter in the second argument of postRecord(). This can be useful if the user is uploading huge files, you can show a progress bar.

Here's an example demonstrating how you can upload files using Skapi:

html

<form
    onsubmit="skapi.postRecord(event, {
    table: {
        name: 'my_photos',
        access_group: 'authorized'
    },
    progress: (p)=>console.log(p) 
}).then(rec=>console.log(rec))"
>
    <input name="description" />
    <input name="picture" multiple type="file" />
    <input type="submit" value="Submit" />
</form>

The name attribute of the file input element will serve as the key name of the file data. Regarless the file input is multi or single, the file(s) will ALWAYS be uploaded as an array of BinaryFile object under the key name picture(the name of the file input element) in the bin key of the RecordData as shown below:

// record data
{
    record_id: '...',
    ...,
    bin: {
        picture: [
            {
                access_group: 'authorized',
                filename: '...',
                url: 'https://...',
                path: '.../...',
                size: 1234,
                uploaded: 1234
                getFile: () => {...};
            },
            ...
        ]
    }
}

The bin data will contain array of BinaryFile objects. This process is handled seamlessly without any complicated file handling required.

Once the files are uploaded, Skapi serves the files using a CDN with no additional setup required.

Every upload is signed for the size it declares

Skapi asks storage for a one file upload permit, and that permit is issued for exactly the byte size the request declares. A request that then sends a different number of bytes is refused by storage and nothing is stored, so the size a file is counted and billed at is always the size it really has.

The SDK declares the true size of the file it sends, encrypted files (the size of the encrypted bytes) and empty files included, so this needs nothing from you. It matters only if you build the upload request yourself instead of using postRecord().

A refused upload does not fail the record. postRecord() resolves as usual and that one file is simply never stored, so fetch the record again when you need to know which files it really holds.

DANGER

If the file is uploaded in a record where the access group is not 'public', the URL value in the BinaryFile objects can expire for security reasons.

Progress Information

When uploading files via postRecord() method, you can attach a ProgressCallback in the progress parameter when uploading files. The ProgressCallback will trigger whenever there is a byte loaded to/from the backend.

let progressCallback = (p) => {
    if (p.status === "upload" && p.currentFile) {
        console.log(`Progress: ${p.progress}%`);
        console.log("Current uploading file:" + p.currentFile.name);
    }
};
skapi.postRecord(someData, {
    table: { name: "my_photos", access_group: "authorized" },
    progress: progressCallback,
});

Downloading Files

To download files from the record, you can use the getFile() method on the BinaryFile object in the record.

Below is an example of how you can download a file from a record:

skapi.getRecords({ record_id: "record_id_with_file" }).then((rec) => {
    let record = rec.list[0]; // record with files attached.

    /*
    // record
    {
        table: {
            name: 'my_photos',
            access_group: 'authorized'
        },
        record_id: '...',
        ...,
        bin: {
            picture: [
                {
                    access_group: 'authorized',
                    filename: '...',
                    url: 'https://...',
                    path: '.../...',
                    size: 1234,
                    uploaded: 1234
                    getFile: () => {...};
                },
                ...
            ]
        }
    }
    */

    let fileToDownload = record.bin.picture[0]; // get the file object from the record

    fileToDownload.getFile(); // browser will download the file.
});

INFO

Uploaded files follow the access restrictions of the record. User must have access to the record in order to download the file.

getFile() allows you to download the file in various ways:

blob: Downloads the file as a Blob object.
base64: Downloads the file as a base64 string.
endpoint: If the file access requires authentication or needs token update, you can request an updated endpoint of the file.
download (or omitted): Triggers file download from the web browser.
text: Downloads the file as text string.
info: Returns file information.

The getFile() method on the BinaryFile object takes two arguments:

dataType: Type of download - blob, base64, endpoint, text, info or download. Defaults to download.
progress: Optional progress callback function. Useful when downloading large files as blob to show progress bar. (Will not work with endpoint or download types.)

Alternatively, you can call the standalone skapi.getFile() method with the file's endpoint URL: skapi.getFile(url, config?). Here url is the file's endpoint URL and config is an optional object that holds dataType (same values as above), progress (the progress callback), expires (use a URL that expires in the given number of seconds; useful for private files), and browserCache / refresh (see Caching Expiring Files below).

If the file has private access restriction, you must use the endpoint type to get the file endpoint URL. The endpoint URL will be a signed URL that can expire after a certain amount of time.

If the file is an image or a video, you can use the url on img tag or video tag to display the file.

Encrypted files are different

None of the above applies to a file attached to a record whose data is encrypted. Its URL serves ciphertext, so endpoint returns something no img or video tag can display, and a plain link downloads unreadable bytes.

Use getFile() with blob, base64, text or download instead: those decrypt. You can tell the two apart without fetching anything, because the bin entry says so:

if (file.encrypted) {
    const blob = await file.getFile('blob');   // decrypted
    imgEl.src = URL.createObjectURL(blob);     // instead of file.url
}

Below is an example of how you can get the endpoint URL of the access restricted private file (The user must have private access granted.):

fileToDownload.getFile("endpoint").then((url) => {
    console.log(url); // endpoint of the file. https://...
});

Below is an example of how you can download a file as a blob, base64 with progress callback:

let progressInfo = (p) => {
    console.log(p); // Download progress information
};

fileToDownload.getFile("blob", progressInfo).then((b) => {
    console.log(b); // Blob object of the file.
});

fileToDownload.getFile("base64", progressInfo).then((b) => {
    console.log(b); // base64 string
});

Caching Expiring Files

Files in a record whose access group is not public are cached for one week automatically. Reading the same private file twice used to download it twice, because the URL a private file is served under changes on every read and browsers cache by URL. Now the first read downloads it and every read after that is served from the browser's own cache, with no network request at all, for a week.

You do not have to do anything for this. It applies to every getFile() call on a record.bin[...] object:

skapi.getRecords({ record_id: "record_id_with_file" }).then((rec) => {
    let file = rec.list[0].bin.picture[0]; // a private file

    file.getFile("blob"); // downloads it
    file.getFile("blob"); // served from the browser cache, no network
});

To display a private file, take the URL from getFile("endpoint") rather than reading the url property:

let src = await file.getFile("endpoint"); // cached for a week
imgElement.src = src;

INFO

The url property on a bin object is deliberately not the cached URL. It stays the record's own file URL, because that is the string you pass back to remove_bin and deleteFiles, the string the dashboards render, and the one that is safe to store: a cached URL is signed for one user and stops working once it expires. Reading file.url directly works too, it is just not the cached path.

INFO

Files reached through a granted private access key (someone else's restricted file that was shared with you) are not cached, because the URL for those cannot be minted in a cacheable form. They keep working exactly as before.

The rest of this section is for files you fetch by URL yourself with skapi.getFile().

When you request a file with expires, Skapi mints a signed URL, and a signed URL is different every time you ask for one. Browsers cache by URL, so a new URL is always a cache miss: the same unchanged file is downloaded again on every page load. For a chat window or a gallery that shows the same private images repeatedly, that is the whole page's worth of traffic, every time.

browserCache fixes it from the other end. It caches the request that mints the URL, so the same URL comes back and the copy the browser already downloaded stays usable.

skapi.getFile(url, {
    dataType: "endpoint",
    expires: 1200, // the signed URL is valid for 20 minutes
    browserCache: 86400, // reuse it, and the downloaded file, for a day
});

The two numbers do different jobs, and it is normal for browserCache to be much larger than expires:

expires is how long the URL works. Keep it short, so a URL that leaks is useless quickly.
browserCache is how long the file stays available locally. The browser serves it from its own cache without checking the URL again.

Once the browser drops the file from its cache, the next load uses a URL that has since expired and fails. Call getFile() again with refresh: true to mint a working URL:

skapi.getFile(url, {
    dataType: "endpoint",
    expires: 1200,
    browserCache: 86400,
    refresh: true, // ignore the cached URL, mint a new one
});

WARNING

If you overwrite a file at the same path, the browser keeps serving the copy it already has until browserCache runs out. Use refresh: true after replacing a file so the new version is picked up.

INFO

browserCache only does something when expires is also set, and the server caps it at 1 week. It is ignored for public CDN URLs, which are already cacheable without it.

Removing Files

To remove files, use the remove_bin parameter in the config argument of the postRecord() method. When updating a record, you can remove files by passing the remove_bin parameter as an array of BinaryFile objects or the endpoint URLs of the files that need to be removed from the record.

Here's an example demonstrating how you can remove files from a record:

...
let fileToDelete = record.bin.picture[0]; // file object retrieved from the record.
skapi.postRecord(undefined, { record_id: 'record_id_with_file', remove_bin: [fileToDelete] });

If you have the endpoint URL of the file, you can also pass the URL as a string in the remove_bin parameter:

skapi.postRecord(undefined, {
    record_id: "record_id_with_file",
    remove_bin: ["https://..."],
});

If you want to remove all files from the record, you can pass the remove_bin parameter as null:

skapi.postRecord(undefined, {
    record_id: "record_id_with_file",
    remove_bin: null,
}); // removes all files from the record.

WARNING

The file that is targeted for removal should be in the record that you are updating.

TIP

If you remove the record that is holding the files, all files that the deleted record was holding will also be completely removed from the database.

Files attached by another account

A file that the project owner or an admin attaches to another user's record is stored under the record's user, the same way as a file that user attached. The uploader in its file information names the record's user, not the account that attached it. The record's user can delete it by its URL, it moves with the record when the record changes access group, and it is deleted with the record.

Files attached by another account earlier

Files that the project owner or an admin attached to another user's record before files were stored under the record's user keep the old behaviour, and are not moved. They stay under the account that attached them, and their uploader names that account, so you can tell one apart by an uploader that is not the record's user_id. The record's user cannot remove such a file by its URL (only remove_bin: null takes it off the record), it does not move when the record changes access group, and it is not removed from storage when the record is deleted.

Deleting Files by URL

deleteFiles() deletes files by their endpoint URLs without updating anything else on the record. It takes one URL or a list of up to 1000, which may belong to different records, and resolves to the records the files were removed from.

let fileToDelete = record.bin.picture[0]; // file object retrieved from the record.
skapi.deleteFiles({ endpoints: [fileToDelete.url] }).then(records => {
    console.log(records); // updated records
});

The whole list is checked before anything is deleted, so if any file in it is refused, no file is deleted.

A user who is not an admin can delete only the files of their own records, except files attached by another account earlier, and is refused otherwise with:

{
    code: "INVALID_REQUEST";
    message: "The record should be owned by the user.";
}

Files on Records of Other Users

Files are an admin right of their own, separate from a record's data and settings. The project owner and every admin (access groups 90 ~ 99) can attach files to another user's record and delete its files, as long as the record is not private. This includes admins in access groups 90 ~ 98, who otherwise can change only a record's subscription settings.

Attach by passing the files to postRecord() with the record's record_id. The postRecord() call itself follows the update rules, so for admins in access groups 90 ~ 98 it must change nothing else: leave data as undefined and send no other setting that differs from the stored record. A read-only record refuses them with Record is read only.
Delete with deleteFiles(). Admins in access groups 90 ~ 98 cannot remove a file from another user's record with remove_bin, because that is a change to the record itself: postRecord() refuses it with Admins can only change the subscription settings of another user's record (remove_bin differs). The project owner and admins in access group 99 can use either.

// An admin attaches a file to another user's record, changing nothing else on it.
skapi.postRecord(undefined, { record_id: "record_id_of_another_user" }, [
    { name: "picture", file: someFile },
]);

// ...and deletes one of its files.
skapi.deleteFiles({ endpoints: ["https://..."] });

A file attached this way is stored under the record's user, so that user can delete it, it moves with the record and it is deleted with the record. Files attached this way earlier are the exception, see Files attached by another account above.

Files on a private record are locked to its user, for everyone, the project owner and admins included, because private is the only access group whose files may be encrypted. Attaching with postRecord() is refused like any other change to another user's private record:

{
    code: "INVALID_REQUEST";
    message: "Only the owner of a record can move it into or out of the private access group.";
}

and deleteFiles() is refused with:

{
    code: "INVALID_REQUEST";
    message: "Only the owner of a private record can delete its files.";
}

Get File Information

You can use getFile() method to get the file information just from the endpoint URL of the file.

Below is an example of how you can get the file information from the endpoint URL:

let fileUrl = "https://...";
skapi.getFile(fileUrl, { dataType: "info" }).then((fileInfo) => {
    console.log(fileInfo);
    /*
    {
        url: string,
        filename: string,
        access_group: number | 'private' | 'public' | 'authorized',
        filesize: number,   // for an encrypted file this is the PLAINTEXT length
        record_id: string,
        uploader: string,   // user ID the file is stored under: the record's user, apart from files attached by another account earlier
        uploaded: number,
        fileKey: string
    }
    */
});

Handling Files ​

Uploading Files ​

Progress Information ​

Downloading Files ​

Caching Expiring Files ​

Removing Files ​

Deleting Files by URL ​

Files on Records of Other Users ​

Get File Information ​

Handling Files

Uploading Files

Progress Information

Downloading Files

Caching Expiring Files

Removing Files

Deleting Files by URL

Files on Records of Other Users

Get File Information