Skip to main content

Large file upload module

The large file upload module transfers files from a directory in the local filesystem to tapio. With this module you can upload any binary file. Data uploaded to tapio via this module will be transferred directly to the configured applications. The data is not persisted in our historic data store and therefore cannot be retrieved via the Historic Data API. The application will only retrieve the Generic data once.

Large file upload module FileCollectorConfig

<Module xsi:type="FileCollectorConfig">
<Id>FileCollectorModule_1</Id>
<BaseDirectory>%TEMP%\tapio\CloudConnector\fileupload</BaseDirectory>
<CheckIntervalInSec>60</CheckIntervalInSec>
<Directories>
<Directory>
<Name>Cnc-4711</Name>
<TapioMachineId>e04df252-f9cd-4f6e-853b-dfe2881bc231</TapioMachineId>
<SourceKey>Spindle.Health</SourceKey>
<Mode>UploadAllExceptLatest</Mode>
<MaxFileSizeForUploadInKb>25000</MaxFileSizeForUploadInKb>
<MaxAgeOfFilesInHours>192</MaxAgeOfFilesInHours>
<FilePattern>*.zip</FilePattern>
<FileNameHandling>UniqueFilenameWithConfirmation</FileNameHandling>
</Directory>
<Directory>
<Name>Saw-Cutter</Name>
<TapioMachineId>e04df252-f9cd-4f6e-853b-dfe2881bc231</TapioMachineId>
<SourceKey>Engine.Health</SourceKey>
<Mode>FileMarkedAsReady</Mode>
<MaxFileSizeForUploadInKb>25000</MaxFileSizeForUploadInKb>
<MaxAgeOfFilesInHours>192</MaxAgeOfFilesInHours>
</Directory>
</Directories>
<Cleanup>
<IntervalInSec>3600</IntervalInSec>
<DeleteAllUnknownFolders>true</DeleteAllUnknownFolders>
<DeleteOldFiles>true</DeleteOldFiles>
<MinFreeSpaceInGb>5</MinFreeSpaceInGb>
</Cleanup>

<Destinations>
<Destination>BatchUpload</Destination>
</Destinations>
</Module>

Inside the configuration there are the following options:

EntryTypeDescription
IdstringUnique identifier of the module.
BaseDirectorystringBase directory for file uploads for this module. CloudConnector needs write and read rights for the directories. The CloudConnector supports also the System-Environment variables. Directory will be created if it doesn't exist.
CheckIntervalInSecintegerCyclical check if there are files to upload Minimum: 15 sec. Default 30 sec. The files are uploaded sequentially.
Directorieslist of DirectoriesList of directories below the base directory that should be monitored.
Cleanupobject of CleanupSpecify the clean up of the configured directories to be sure the memory will not run full.
Destinationslist of DestinationsList of destination with modules which are doing the real file upload to tapio.

Directories

Directories contains a list of defined directories that are monitored by the CloudConnector. If there is a file that matches the pattern, that file is uploaded.

EntryTypeDescription
NamestringThe name of the directory.
TapioMachineIdstringThe tapio machine id for this subscription group, all data received via this group will be tagged with the configured tapio machine id.
SourceKeystringKey that is used to identify the data in tapio.
ModestringMode when the file will be uploaded. UploadAllExceptLatest, FileMarkedAsReady, default: UploadAll.
MaxFileSizeForUploadInKbintegerMax. file size. If the file is larger it will be deleted (and upload fails).
MaxAgeOfFilesInHoursintegerMax. age of files before they get deleted, even if not uploaded.
FilePatternstringOpt.: Restrict to specific file pattern. Default value is *.*.
FileNameHandlingstringUniqueFilenameWithConfirmation: Uses the original file name for upload and creates confirm or error files. CC creates a {filename}.confirm or {filename}.error file for success / failure. These files are also automatically deleted with the "max-age" parameter. Filenames must be unique. GenerateUniqueFilename: Generates a unique filename on upload. No confirm or error files are created. default: UniqueFilenameWithConfirmation
CompressFilesBeforeUploadbooleanIf set to true the files will be zip compressed before they are uploaded. Default false.

Mode

This module supports three different modes to handle the upload of the files in the configured directory.

ModeDescription
UploadAllUpload all files in the directory.
UploadAllExceptLatestUpload all files except the newest file. Files are ordered by last write.
FileMarkedAsReadyOnly upload the files which filenames end with *.ready.

Cleanup

This section is needed to define how the cleanup of the directory should be done. This is also carried out when the CloudConnector is not onboarded by a customer (tapio deactivated state) or has no connection to tapio.

⚠ If the CloudConnector (Windows or Linux service) is disabled/deactivated, there is no cleanup. Be aware that this can lead to full disks ⚠.

EntryTypeDescription
IntervalInSecstringCleanup interval in seconds. Default 3600 sec.
DeleteAllUnknownFoldersstringDeletes all folders below the Base Directory which are not configured in the Directories list. Default true.
DeleteOldFilesbooleanEnable/disable deletion of old files. Disabling could lead to full disks if machine is not onboarded. Default true.
MinFreeSpaceInGbdoubleIf the space requirement is less, files are deleted starting with the oldest files in all directories until the space requirement is reached. So you might loose data. Default 3.0 Gb.

Destinations

EntryTypeDescription
DestinationstringModule Id of a module which uploads the files. This module must be of type BatchUploadConfig.

If the destination is not set correct the module can not upload the files to tapio.