Flat file source not communicating with script component properly

Hi there,

I have a flat file source in my data flow that is suppose to feed through a couple of columns to a script component. All of the columns have data but when I debug in my script component (with the aid of messageboxes) I can see that one of then is empty.

This is wrong because when I preview that data in the flat file source, I can see that the columns are all populated.

Has anyone ever encountered that a flat file source doesn't communicate with a script component properly? I mean it does pass through the data of all other columns except for one.

I have replaced the components in case it had become corrupt and I even rebuild the package but the problem still persists.

Any advice would be welcome

Regards
Mike

Put data viewer between the Flat File Source and the Script component and check what data actually enters your script.

HTH,

Bob

|||Thanks for the reply,

I put a viewer on and saw that the script gets all the populated columns except for the one I mentioned earlier. Thus the problem seem to be with the flat file source not passing the data in that column through to the script.
But why would it do that even though it shows all of the columns populated when I click on the preview button on the flat file source?

Regards

**EDITED**

I have also noted that when I check the "retail null values from row as null values in data flow" checked box, then it does populate all columns correctly but I cant have that checked box checked since it does not go to the correct path on my workflow with the nulls. When I uncheck it, it misses the one column again but not the other ones. This is puzzling me very much
|||

What version of the SSIS do you have installed (RTM, SP1, SP2)? I think there was a flat file parsing bug fixed in one of SP releases that would potentialy cause this behavior.

Thanks,

Bob

|||Hi Bob,

I have recently (last week) upgraded to service pack 2 and also updated my ssis from Version 9.00.20(something) to Version 9.00.3042.00.

The package was built on an older version and pre service pack 2. I wonder if that is not the case here.

Thanks for the reply

Regards
Mike
|||

Would it be possible for you to isolate a few rows that fail, create another simple file with only those two rows, obfuscate your data and post your file here?

Just make sure the newly created file still fails the same way as the original one.

Thanks,

Bob

|||Hi Bob,

Just for some extra clarity, none of the rows are actually failing. They all pass through successfully except for one of the columns (SiteCode) in the file does not get read properly by the flat file source. It does not fail or warn me about anything, it just doesn't detect the data in this specific column. It passes an empty string through for all rows in that column.

Like I mentioned before, the funny thing is that when I preview the file in the flat file source in the data flow task, it does actually show that the column is populated. All of the SiteCodes are populated in the preview window but that preview data just doesn't get passed through to the other data flow component.

Would it still be usefull for me to post the file?

Regards
Mike
|||

Yes, by "failing" I meant not being parsed properly.

If you can provide the sample of data I can try to debug it here and eventually find out what is causing the behavior you are seeing.

Thanks,

Bob

|||Hi,

I hope this will be sufficient
The column with the 'BAL' data doesn't get read properly
Regards

FMC|FMCG|22110|FMCG Warh|BAL|Balance Sheet|BAL|Balance Sheet|22110|FMCG Warehouse Balance sheet|10030||Cost - Comp Equip 3yrs|2006|10|144.00|AA|General Ledger||
FMC|FMCG|22110|FMCG Warh|BAL|Balance Sheet|BAL|Balance Sheet|22110|FMCG Warehouse Balance sheet|10050||Cost - Comp Equip 10yrs|2006|10|233.00|AA|General Ledger||

|||

I tried to parse these two rows using Flat File Source and could not see any problem with it.

Could you tell me again what is not read properly? I have seen "BAL" values read in both; 5th and 7th columns.

|||Hi Bob,

I managed to find a workaround on this problem. When I click the "retain null values from source as null values in data flow" check box, the data does get passed on correctly but to the wrong exit point (error exit point).

I did further investigation as to why it would go to the error exit point and realized that it actually picks up on a null value and does not know how to handle it. I got the "object reference not set as an instance of a ...etc" error.

Adding code to check for these nulls and setting it to dbnull.value helped with my problem and things are working fine now. I also believe that my problem might also be because we did not use text qualifiers in this project but this is only speculation.

From now on I will know what to do with custom scripts reading and processing text files so that I don't leave the system open for these possibly unhandled situations.

Regards

Flat file source not communicating with script component properly

Hi there,

I have a flat file source in my data flow that is suppose to feed through a couple of columns to a script component. All of the columns have data but when I debug in my script component (with the aid of messageboxes) I can see that one of then is empty.

This is wrong because when I preview that data in the flat file source, I can see that the columns are all populated.

Has anyone ever encountered that a flat file source doesn't communicate with a script component properly? I mean it does pass through the data of all other columns except for one.

I have replaced the components in case it had become corrupt and I even rebuild the package but the problem still persists.

Any advice would be welcome

Regards
Mike

Put data viewer between the Flat File Source and the Script component and check what data actually enters your script.

HTH,

Bob

|||Thanks for the reply,

I put a viewer on and saw that the script gets all the populated columns except for the one I mentioned earlier. Thus the problem seem to be with the flat file source not passing the data in that column through to the script.
But why would it do that even though it shows all of the columns populated when I click on the preview button on the flat file source?

Regards

**EDITED**

I have also noted that when I check the "retail null values from row as null values in data flow" checked box, then it does populate all columns correctly but I cant have that checked box checked since it does not go to the correct path on my workflow with the nulls. When I uncheck it, it misses the one column again but not the other ones. This is puzzling me very much
|||

What version of the SSIS do you have installed (RTM, SP1, SP2)? I think there was a flat file parsing bug fixed in one of SP releases that would potentialy cause this behavior.

Thanks,

Bob

|||Hi Bob,

I have recently (last week) upgraded to service pack 2 and also updated my ssis from Version 9.00.20(something) to Version 9.00.3042.00.

The package was built on an older version and pre service pack 2. I wonder if that is not the case here.

Thanks for the reply

Regards
Mike
|||

Would it be possible for you to isolate a few rows that fail, create another simple file with only those two rows, obfuscate your data and post your file here?

Just make sure the newly created file still fails the same way as the original one.

Thanks,

Bob

|||Hi Bob,

Just for some extra clarity, none of the rows are actually failing. They all pass through successfully except for one of the columns (SiteCode) in the file does not get read properly by the flat file source. It does not fail or warn me about anything, it just doesn't detect the data in this specific column. It passes an empty string through for all rows in that column.

Like I mentioned before, the funny thing is that when I preview the file in the flat file source in the data flow task, it does actually show that the column is populated. All of the SiteCodes are populated in the preview window but that preview data just doesn't get passed through to the other data flow component.

Would it still be usefull for me to post the file?

Regards
Mike
|||

Yes, by "failing" I meant not being parsed properly.

If you can provide the sample of data I can try to debug it here and eventually find out what is causing the behavior you are seeing.

Thanks,

Bob

|||Hi,

I hope this will be sufficient
The column with the 'BAL' data doesn't get read properly
Regards

FMC|FMCG|22110|FMCG Warh|BAL|Balance Sheet|BAL|Balance Sheet|22110|FMCG Warehouse Balance sheet|10030||Cost - Comp Equip 3yrs|2006|10|144.00|AA|General Ledger||
FMC|FMCG|22110|FMCG Warh|BAL|Balance Sheet|BAL|Balance Sheet|22110|FMCG Warehouse Balance sheet|10050||Cost - Comp Equip 10yrs|2006|10|233.00|AA|General Ledger||

|||

I tried to parse these two rows using Flat File Source and could not see any problem with it.

Could you tell me again what is not read properly? I have seen "BAL" values read in both; 5th and 7th columns.

|||Hi Bob,

I managed to find a workaround on this problem. When I click the "retain null values from source as null values in data flow" check box, the data does get passed on correctly but to the wrong exit point (error exit point).

I did further investigation as to why it would go to the error exit point and realized that it actually picks up on a null value and does not know how to handle it. I got the "object reference not set as an instance of a ...etc" error.

Adding code to check for these nulls and setting it to dbnull.value helped with my problem and things are working fine now. I also believe that my problem might also be because we did not use text qualifiers in this project but this is only speculation.

From now on I will know what to do with custom scripts reading and processing text files so that I don't leave the system open for these possibly unhandled situations.

Regards

FK Checks

Suppose you have two (or more) tables with foreign key constraints. My
question is thus:

Is it better to check if the fk exists before you try to perform the
insert or let SQL do it for you?

On one hand, if you check yourself and the key does not exist you can
gracefully handle it (maybe exit out of method with error). If you let
SQL do it, the server will throw an error which cannot be suppressed.

On the performance side, you doing the check will incur a slight (VERY
slight) hit since SQL will ALSO check anyways.Jason (JayCallas@.hotmail.com) writes:
> Suppose you have two (or more) tables with foreign key constraints. My
> question is thus:
> Is it better to check if the fk exists before you try to perform the
> insert or let SQL do it for you?

Depends on business requirements. Basically, Fkeys is the database's
mean of protection against bad data. If the user interface does not
perform any checks itself, and relies on the database, you may avoid
integrity violations, but the users may not get adequate error message.

The way I see database constraints constitutes an inner defense line.
The user interface should help the user, and have its own defense line.
Error messages from the database exposed to the user, should be considered
a bug.

But validation in a GUI, can be quite different from the validation in
SQL Server. Normally you don't let the user to type in the FK as free-
text, but you let him choose from a drop-down box or from a search screen.
Once you know, the user has selected data this way, you can assume that
you are safe. (Although, someone may delete the row before the users
saves.)

--
Erland Sommarskog, SQL Server MVP, sommar@.algonet.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techin.../2000/books.asp|||Erland Sommarskog <sommar@.algonet.se> wrote in message news:<Xns944C1D0487E5Yazorman@.127.0.0.1>...
> Depends on business requirements. Basically, Fkeys is the database's
> mean of protection against bad data. If the user interface does not
> perform any checks itself, and relies on the database, you may avoid
> integrity violations, but the users may not get adequate error message.
> The way I see database constraints constitutes an inner defense line.
> The user interface should help the user, and have its own defense line.
> Error messages from the database exposed to the user, should be considered
> a bug.
> But validation in a GUI, can be quite different from the validation in
> SQL Server. Normally you don't let the user to type in the FK as free-
> text, but you let him choose from a drop-down box or from a search screen.
> Once you know, the user has selected data this way, you can assume that
> you are safe. (Although, someone may delete the row before the users
> saves.)

I should have been more explicit in my question. I was referring to
performing checks within stored procedures.

Assuming Table2 has a FK relationship with Table1.

IF EXISTS (SELECT * FROM Table1 WHERE Table1.Field1 = @.field1)
INSERT INTO Table2 (@.field1, @.field2, field3)

as opposed to

INSERT INTO Table2 (@.field1, @.field2, field3)
IF @.@.Error RAISERROR('Unable to insert into Table2 because of missing
constraint', 15, 1)|||Jason (JayCallas@.hotmail.com) writes:
> I should have been more explicit in my question. I was referring to
> performing checks within stored procedures.
> Assuming Table2 has a FK relationship with Table1.
> IF EXISTS (SELECT * FROM Table1 WHERE Table1.Field1 = @.field1)
> INSERT INTO Table2 (@.field1, @.field2, field3)
> as opposed to
> INSERT INTO Table2 (@.field1, @.field2, field3)
> IF @.@.Error RAISERROR('Unable to insert into Table2 because of missing
> constraint', 15, 1)

Again it depends. Say that your procedure is to be called from a GUI. In
this case you can presume that the GUI gets the information from the
databaes, and a foreign-key violation could only occur if the GUI is
incorrect. On the other hand, say that you are accepting data from an
external source over which you have no control. In this case, it is better
to check explicitly, so that you can log incorrect data appropriately.
Recall that when a constraint blows up, you cannot in T-SQL determine
which constraint that fired.

--
Erland Sommarskog, SQL Server MVP, sommar@.algonet.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techin.../2000/books.asp