Questions tagged [amazon-s3]

Amazon S3 (simple storage service) is an online object storage service from Amazon Web Services. QUESTIONS MUST BE ABOUT PROGRAMMING. Questions about general S3 support, functionality, configuration, etc. are OFF-TOPIC.

Filter by
Sorted by
Tagged with
-1
votes
0answers
9 views

Session-django for access username from mysql database

I want to access mysql database of django and by taking username from database I want to create username's directory on s3 and by using django-sessions..how to do this?
1
vote
0answers
11 views

laravel-ffmpeg - fopen(/tmp/laravel-ffmpegi340VY.mkv): failed to open stream: No such file or directory

Using the laravel-ffmpeg librabry to convert video file from .webm to mp4 aws s3 bucket url and using following code. $videoFile = '/courses/images/1579626678msr-2020-01-21T17-11-18-771Z.webm'; //= ...
0
votes
0answers
7 views

Unable to upload large files via Strapi

I need to be able to upload large video files ~2GB to Amazon S3 via Strapi (3.0.0-beta.18.4) but the uploads fail. Uploading small files works fine. The upload fails with the following 500 internal ...
0
votes
0answers
22 views

passing parameters from models

I have this model where I upload a file and save it to s3 . Here is the model. class ExcelFiles(models.Model): PLAN_CHOICES = ( ('x1', 'x1'), ('x2', 'x2'), ('x3', 'x3') ...
0
votes
1answer
19 views

Unable to access aws s3 from docker container. ECONNREFUSED

I'm trying to access aws s3 from a docker container. IAM role with AWSS3FullAccess is created and credentials are set using json config file. Inside the code reading config as AWS.config....
0
votes
2answers
18 views

Is it required to provide access_key and secret_key in order to access object from s3 in lambda function?

I want to write a lambda function that would update object file in the s3 bucket. I am using same account and region in order to implement that. As per the policy, the secret key and access key that ...
0
votes
0answers
9 views

Is there any cli tool that can buffer piping stdin data so that it writes to s3 in date partitioned files?

We're currently using kinesis with autoscaling to write to s3. If someone makes a mistake that pipes way too much data, we get hit by a very expensive bill. This seems like overkill. Is there a cli ...
1
vote
0answers
20 views

Data protection key ring with S3 not working properly in dot net core

We are using MVC with dotnet core 2.2.0 version. For data protection we are using AWS S3 as persistent storage. Web app is hosted as windows service on two machines behind the load balancer. ...
0
votes
1answer
6 views

Error in joblib.load when reading file from s3

When trying to read a file from s3 with joblib.load() I get the error ValueError: embedded null byte when attempting to read files. The files were created by joblib and can be successfully loaded ...
1
vote
0answers
9 views

AWS Glue table partitions for data stored in multiple buckets

I have a Glue table with 10s of thousands of partitions with the following partition keys: "PartitionKeys": [ { "Name": "thing", "Type": "string" }, { "Name": "...
0
votes
0answers
12 views

Send go test coverage output to stdout - send it to S3

I want to send the html for go test -c to stdout, so I can serve the resulting HTML from S3 static assets server. Something like this: arti_fact="s3://cm-html/cm-api/$commit_sha" go test -...
0
votes
0answers
5 views

AWS Glue- job runs with no errors, but output is not displayed

I have data in AWS MySQL RDS and the requirement is to grab data from a table to the csv file and place it in S3. To achieve that i am using AWS glue and have code as shown below. The job runs with no ...
-1
votes
0answers
16 views

Upload multipart/form-data to S3 from lambda (Nodejs)

I want to upload multipart/form-data file to S3 using Nodejs. I have tried various approaches but none of them are working. I was able to write content to S3 from lambda but when the file is ...
0
votes
1answer
20 views

Access S3 bucket without running aws configure with kubernetes

I have an S3 bucket with some sql scripts and some backup files using mysqldump. I also have a .yaml file that deploys a fresh mariadb image. As I'm not very experienced with kubernetes yet, if I ...
0
votes
0answers
28 views

JavaScript Promise not returning body in response

I am running the following handler in AWS Lambda with Node 12.x. This handler creates a pdf from a given URL and stores the pdf in an s3 bucket. I am trying to return some data in the body of the ...
0
votes
0answers
11 views

Querying CSV file in S3 through Athena throws 403 Erroe Access Denied

I have a folder created in S3 called rawdata. There are Subfolders inside rawdata called masters, There are CSV files uploaded in rawdata and msters folders. I have created Tables in AWS Athena. While ...
0
votes
1answer
16 views

Possible to supply wildcard to aws S3 batch job

I want to run an S3 batch job. To do that S3 requires a manifest (*.csv) telling it what objects to iterate over. Well my bucket has 1.2 billion objects in it. But, I don't want to make a CSV file ...
0
votes
1answer
26 views

Setting up AWS S3 bucket permissions

I have an s3 bucket (e.g. mybucket) which currently has permissions set as follows: Block all public access On | |_Block public access to buckets and objects granted through new access control ...
0
votes
1answer
13 views

Django Model FileField storage use class depends on settings

I would like to manage django multimedia files locally and in the aws s3, I have all the configuration and if I specify the storage class in the model it works fine, but it is not practical because I ...
0
votes
0answers
17 views

Gzip file is getting corrupted while uploading it into S3 bucket using Java application

I am uploading a gzip file into S3 bucket using java application, the data of which will be used in Athena. The gzip file is getting corrupted while uploading. Due to which Athena is unable to view ...
0
votes
0answers
15 views

How can I write the output inside the folder of a S3 bucket using Lambda fuction

I am coding in Python Lang. in AWS Lambda and I need to write the output which shows in output logs in a folder of S3 bucket. The code is, client = boto3.client('s3') ouput_data = b'Checksums are ...
0
votes
1answer
13 views

AWS s3 and minio incompatibility signature checking if url params contains ;

I have a problem with incompatibility between minio(RELEASE.2019-07-17T22-54-12Z) and AWS S3 I request the generation of pre-signed URL for download file from minio From minio I got pre-signed URL ...
0
votes
1answer
20 views

Storing file from request stream in node.js

I want to create Web API, where, I am uploading audio stream from the client-side and in the nodejs server, I want to save the audio stream as a file an then I wan to upload the file to S3. How Can I ...
1
vote
1answer
65 views

Out of memory when downloading 460mb file from S3 to AWS lambda using go aws sdk

I'm running lambda using the aws-sdk-go-v2 but running into memory issues when downloading larger files. I've tried all sorts of combinations of partSize and concurrency but I either hit the timeout ...
0
votes
2answers
27 views

AWS: How to transfer files from ec2 instance (Windows Server) to S3 daily?

Can someone explain me whats the best way to transfer data from a harddrive on an EC2 Instance (running Windows Server 2012) to an S3 Bucket for the same AWS Account on a daily basis? Backround idea ...
0
votes
0answers
26 views

How to upload file to aws s3 using php rest api?

I am new to aws server, I am writing php rest api for uploading image into aws s3 bucket, but I am not able to upload it. I am sending image as base64_encode format and decoding it before uploading. ...
0
votes
0answers
14 views

cannot read data on S3 with pySpark deployed on EC2 cluster

I set up pySpark on my EC2 cluster, the version is http://mirrors.koehn.com/apache/spark/spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop3.2.tgz I can see my bucket with boto3 When I tried to read ...
0
votes
0answers
25 views

ValueError: Filename must be a string while uploading file to s3 bucket

I am sending an excel file as a request in postman and need to upload this to s3 . I access the file from request and send it to s3. @api_view(['POST']) def excel_upload(request): print("request",...
0
votes
0answers
11 views

Uploading multiple Folders to the Amazon S3 Bucket

I am writing a Powershell script to upload multiple folders and sub folders to the AWS S3 bucket, it is giving the below error. I recreated the same for uploading a single folder and the error remains ...
0
votes
1answer
25 views

Dynamic table creation SQLAlchemy ORM

I am new to SQLAlchemy ORM. I am trying to build a AWS S3 ingestion program which will ingest any CSV file from S3 bucket to Postgres through ORM. I am trying to read the first row of the CSV file and ...
0
votes
1answer
14 views

Files in s3 bucket are automatically moving to tombstone folder

I have created an S3 bucket, not sure what am I missing with IAM lifecycle policies. Files in s3 bucket are automatically moving to tombstone folder after few days. how to stop this? I have enabled ...
0
votes
0answers
13 views

how do I get koa to serve an image that has been retrieved from s3 via getObject?

I'm trying to write my own image resize server (because the world definitely needs another) based off of the example found at https://aws.amazon.com/blogs/networking-and-content-delivery/resizing-...
0
votes
0answers
25 views

Getting “server keep idle connection” errors using MinIO with Go as I open many connections in a row, how can I solve the issue?

I have a service which wants to open 28 connections to different S3 objects to save them there for later reuse. At this time, I'm testing my code against localstack to make sure it will work and ...
0
votes
0answers
20 views

Corrupted Data with Spark Application using s3a

I am trying to download some files from S3 in a Spark Job using the aws hadoop connector. I see that a few of those files are corrupted and have the following content as the first line: read tcp read:...
0
votes
0answers
15 views

Presto fails to import PARQUET files from S3

I have a presto table that imports PARQUET files based on partitions from s3 as follows: create table hive.data.datadump ( tUnixEpoch varchar, tDateTime varchar, temperature varchar, ...
0
votes
0answers
15 views

Python Boto3 PutBucketReplication operation: The XML you provided was not well-formed or did not validate against our published schema

I'm trying to update the bucket's replication rule using python boto3. But I keep getting the error: botocore.exceptions.ClientError: An error occurred (MalformedXML) when calling the ...
1
vote
0answers
35 views

Can this PHP script be ran without re-directing the user to the homepage?

I am building a wordpress site that will act as a replacement for dropbox. We are using an S3 bucket as our server to host uploaded images. On the site I am using the 'php everywhere' plugin to put ...
0
votes
0answers
22 views

How to upload url image to s3 with boto3

Given a url of an image i want to upload the image to s3 bucket with boto3. I have seen ways to do this using the requests module but since i am using lambda im wondering if i could do it with off the ...
0
votes
1answer
23 views

Generating different signed urls with a loop (S3)

im having an issue trying to create a list of urls for multipart upload to S3 using lamba, im trying to generate the URLs using a loop in the lambda function, but it only generates 1 and keeps ...
0
votes
1answer
17 views

Laravel download from cloud storage throws file not found error

I have a file stored in an s3 bucket. I can upload fine but when I try to download using the code below I get file not found exception thrown e.g. Symfony\Component\HttpFoundation\File\Exception\...
0
votes
1answer
20 views

Triggering lambda from s3 vs Triggering lambda from sqs

I am working on a requirement where you need to submit forms into the RDS database. The forms should not be greater than 20KB. There can be spikes for form submission and we might need to support 300,...
0
votes
1answer
23 views

Loading S3File in AWS

I'm trying to download my cifar 10 data that is in S3 to train it in AWS SageMaker. I'm using this code to load the data: import s3fs fs = s3fs.S3FileSystem() def unpickle(file): dict = pickle....
0
votes
0answers
7 views

Authenticate to S3 using External IdP

So I hope there is a very simple answer to this. I have setup a bucket on S3 and there are some files there. Here is the flow I am looking for: Give link to a file on the bucket to another person ...
1
vote
0answers
42 views

How to restrict outgoing network access from EC2 instance?

I am contracting out some ML tasks. I want all of the work the contractor does to be on a machine 'owned' by me without me buying them a physical machine, so I believe I want to spawn an EC2 instance (...
0
votes
0answers
7 views

Evaporate js: handle aws key expiration

In our case, aws_key is only valid for 1 hour, so I am trying to understand how to handle the situation when the upload process has been started with Evaporate#add using one accessKey, but URL was ...
0
votes
0answers
12 views

Amazon s3 Storage upload object using Presigned url android

I am working on Amazon S3 client I want to upload an object using presigned URL. But when I tried to upload a file using Retrofit or other utility libraries it uploaded successfully but corrupt the ...
1
vote
0answers
21 views

Spark java.io.IOException: Failed to rename S3AFileStatus

I'm writing spark (2.4.4) dataframe to S3 and get sporadic java.io.IOException: Failed to rename S3AFileStatus errors. Spark is unbundled installed over hadoop 3.1.3. The error is known and caused by ...
2
votes
2answers
17 views

How to set multiple aws credentials in nodejs aws-sdk module?

I need to use multiple AWS credentials for different services like s3, SNS.... var awsS3 = require('aws-sdk'); var awsSes = require('aws-sdk'); awsS3.config.update({ region: config.awsRegion, ...
0
votes
0answers
15 views

Any way to temporary store and retieve several list of jgp images downloaded from AWS bucket dynamically

Any way to temporary store and retrieve several list of jgp images downloaded from AWS-S3 bucket dynamically like an arrayList where it will be easy to retrieve one by one when needed in code later on ...
0
votes
0answers
18 views

aws-cli installation AWS CLI Version 1 without Sudo request command line developer tools

MacOS Catalina I have the need to install the aws-cli for enduser S3 features. Using the recommended installation with terminal: curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-...