Add Custom User Attributes
Add Custom User Attributes
Use the REST API to upload custom user attributes like org unit, location, country, manager, and so on, to enrich Skope IT events. This is an alternate method to using Directory Importer (formerly know as Active Directory {AD} Importer). While Directory Importer fetches attributes directly from a user directory, like AD or other directory, this method allows you to use data from other sources, like an HR tool or any other IAM. The data from such tools is often more accurate than data from an AD.
The REST API enables you to upload a CSV file with all the custom user attributes directly to the Netskope UI.
Important CSV File Properties
The following are some important factors to consider when using CSV file to import users.
- The file uploaded must be in CSV format with a header row that defines the custom attribute fields.
- The names in the header fields are used for the field names displayed in Skope IT events.
- The first field must be the email ID, which must map to users’ email address.
- Field names should not have spaces or double quotes.
- The custom attributes can be in double quotes, and must be inside double quotes if they have spaces or a comma in the value.
- The first row of the CSV must contain field names which will be used as custom attribute field name.
- Maximum number of custom attributes supported is 15.
- There is no limit on the length of the value for the attributes, but it is recommended that the length be kept as short as possible. Maximum recommended length for the values is 40.
- If the header is in uppercase or mixed case, it will be converted to lowercase in accordance with the other Netskope attributes.
- If there are duplicate key records, the first one will be the one considered. The duplicates are resolved based on the ‘email_id’ field. The email_id field is considered as the unique key field.
- Extensions allowed for the files are .csv and .txt.
- If the CSV file has more than 1000 records, the import script automatically splits the file into 1000 row chunks. The script to add custom user attributes is located at the end of this topic.
Example of a CSV File
EMAIL_ID,USER_ID,COMPANY_CODE,ORGANIZATION_UNIT,COUNTRY_CODE,LOCATION,USER_TYPE "user1@org.com","user1",143,"NorCal","US","SC","Internal" "user2@org.com","user2",143,"NorCal","US","SC","Internal" "user3@org.com","user3",143,"SoCal","US","SD","Internal"
Using the Script
Follow the procedure below to import users using the script.
- Build your CSV file following the guidelines stated and save it to a local directory. XY
- Get your REST API v1 token from the Netskope UI (Settings > Tools > Rest API v1).
- Copy the script to a local directory.
- Run the script as follows:
For fresh import, run the script in two steps, 1 – Delete existing entries & 2 – Import new entries.Delete Existing Entries
./ns_send_adsync.sh -f oldusers.csv -u "https://addon-<tenant-URL>/adsync/z99W4kf7fhh5FgjexPl?token=68fcc5b6edfd88e3c833248a166d6f6d" -o "delete"
Import New Entries
./ns_send_adsync.sh -f newusers.csv -u "https://addon-<tenant-URL>/adsync/z99W4kf7fhh5FgjexPl?token=68fcc5b6edfd88e3c833248a166d6f6d"
Note
It's important you see the message "Data successfully Uploaded to Netskope". Otherwise, the upload was not successful. You can parse the log file and verify it in the Skope IT events.
Sample Usage
./ns_send_adsync.sh -f sample1.csv -u "https://addon-<tenant-URL>/adsync/z99W4kf7fhh5FgjexPl?token=68fcc5b6edfd88e3c833248a166d6f6d" -o "delete"
Sample Output
BaseURL : https://addon-<tenant-URL> Orgkey : z99W4kf7fhh5FgjexPl secureUPN: 0 UploadFilesize=3 MAXCHUNKSIZE=1000 constructing and sending adsync request for : sample1.csv {"config":{"secureUPN":"0"}, "users":[{"action": "delete","email": "user1@org.com","upn": "user1@org.com","enabled":true,"groups":[],"custom_attributes": {"EMAIL_ID": "user1@org.com","USER_ID": "user1","COMPANY_CODE": "143","ORGANIZATION_UNIT": "NorCal","COUNTRY_CODE": "US","LOCATION": "SC","USER_TYPE": "Internal"}},{"action": "delete","email": "user2@org.com","upn": "user2@org.com","enabled":true,"groups":[],"custom_attributes": {"EMAIL_ID": "user2@org.com","USER_ID": "user2","COMPANY_CODE": "143","ORGANIZATION_UNIT": "NorCal,Eternity","COUNTRY_CODE": "US","LOCATION": "SC","USER_TYPE": "Internal"}},{"action": "delete","email": "user3@org.com","upn": "user3@org.com","enabled":true,"groups":[],"custom_attributes": {"EMAIL_ID": "user3@org.com","USER_ID": "user3","COMPANY_CODE": "143","ORGANIZATION_UNIT": "SoCal","COUNTRY_CODE": "US","LOCATION": "SD","USER_TYPE": "Internal"}}], "schema":["EMAIL_ID","USER_ID","COMPANY_CODE","ORGANIZATION_UNIT","COUNTRY_CODE","LOCATION","USER_TYPE"]} {"status":"success","msg":"Successfully submitted data ","errors":[],"warnings":[]}role
Script Listing
#!/bin/bash
# ***********************************************************
# Example Bash script to upload csv files for custom user
# attributes
# Copyright: Copyright (c) 2021, Netskope, Inc
# ***********************************************************
ADSYNCURL=
CSVFILE=
OPERATION="patch"
function usage
{
echo "usage: $0 -u <uiendpoint> -f <path of the csv file to upload> [-o <\"delete\"|\"patch\">] [-h]"
}
function get_schema() {
csvFile=$1
head -1 $csvfile|gawk '
BEGIN {
FPAT = "([^,]+)|(\"[^\"]+\")"
printf("[")
}
{
if (NR==1) {
nfields=NF
for (i=1; i<=NF; i++) {
gsub("\"","",$i)
printf("\"%s\"",tolower($i))
if (i<NF) {
printf(",")
}
}
}
}
END {
printf("]")
}'
}
function construct_user_request() {
action=$1
csvFile=$2
gawk '
BEGIN {
FPAT = "([^,]+)|(\"[^\"]+\")"
count=0
nfields=0
printf("[")
}
{
if (NR==1) {
nfields=NF
for (i=1; i <=NF; i++) {
fileds[i]=tolower($i)
}
}
if (NR > 2) {
printf(",")
}
if (NR>1) {
if(NF==nfields) {
gsub("\"","",$1)
printf("{")
printf("\"action\": \"'$action'\",")
printf("\"email\": \"%s\",",$1)
printf("\"custom_attributes\": {")
for (i=1; i <=NF; i++) {
gsub("\"","",$i)
printf("\"%s\": \"%s\"",fileds[i],$i)
if (i < NF) {
printf(",")
}
}
printf("}")
printf("}")
}
++count
}
}
END {
printf("]")
}' ${csvFile}
}
BASEURL=
ORGKEY=
TOKEN=
secureUPN="0"
function get_secureUPN() {
BASEURL=${ADSYNCURL%/adsync*}
echo "BaseURL : $BASEURL"
ORGKEY=${ADSYNCURL#*/adsync/}
ORGKEY=${ORGKEY%\?*}
echo "Orgkey : $ORGKEY"
TOKEN=${ADSYNCURL#*token=}
TOKEN=${TOKEN%&*}
response=`curl -X GET --silent -k "$BASEURL/adconfig?orgkey=$ORGKEY&capability=1"`
secureUPN=`echo $response|grep '"secureUPN"\s*:\s*"[0-2]"' -o|awk -F':' '{if (NF<2){print "\"0\""}else{print $2}}'`
secureUPN=${secureUPN#*\"}
secureUPN=${secureUPN%\"*}
if [ -z "${secureUPN}" ]; then
echo "NOTICE : Couldn't get secureUPN config from adconfig, Setting it to \"0\" "
secureUPN="0"
fi
echo "secureUPN : $secureUPN"
}
function construct_and_send_request() {
action=$1
csvFile=$2
#construct_user_request "create" $CSVFILE
users=`cat $csvFile|construct_user_request $action`
schema=`cat $csvFile|get_schema`
request="{\"config\":{\"secureUPN\":\"$secureUPN\"}, \"users\":$users, \"schema\":$schema}"
echo $request
ADSYNC_RESPONSE=`curl -k --silent -X POST \
-H 'Content-Type:application/json' \
-H 'UserAgent:Netskope Adapters -v78.0.0.111' \
-H 'Accept:application/netskope.adsync.v8' \
-d "$(echo $request)" \
-k "$ADSYNCURL"`
echo $ADSYNC_RESPONSE
}
function uploadFile() {
UPLOADFILE="$1"
OPERATION=$2
declare -i UPLOADFILESIZE=`tail -n +2 ${UPLOADFILE}|wc -l| awk '{print $1}'`
echo "UploadFilesize=$UPLOADFILESIZE"
# MAXNUMOFUSERSPERREQ
declare -i MAXUPLOADSIZE=1000
echo "MAXCHUNKSIZE=$MAXUPLOADSIZE"
if [ ${UPLOADFILESIZE} -ge ${MAXUPLOADSIZE} ]; then
UPLOADCHUNKFILE=`echo chunked_${UPLOADFILE}`
echo "CSV file contains more than $MAXUPLOADSIZE records. splitting......"
echo "Chunked File that will be uploaded is $UPLOADCHUNKFILE"
if [ -d "./nsuploadtmp" ]; then
echo 'Deleting the temporary nsuploadtmp directory'
rm -rf ./nsuploadtmp
fi
# create a working dir
mkdir -p nsuploadtmp
tail -n +2 $UPLOADFILE|split -l $MAXUPLOADSIZE - --filter='sh -c "{ head -n 1 '$UPLOADFILE'; cat; } > $FILE"' nsuploadtmp/ns_uacsv_
FILELIST=`ls nsuploadtmp/ns_uacsv_*`
for f in $FILELIST
do
echo "constructing and sending adsync request for : $f"
construct_and_send_request $OPERATION $f
done
else
echo "constructing and sending adsync request for : $UPLOADFILE"
construct_and_send_request $OPERATION $UPLOADFILE
fi
}
OPERATION="patch"
while [ "$1" != "" ]; do
case $1 in
-u | --uiendpoint ) shift
ADSYNCURL=$1
;;
-f | --csvfile ) shift
CSVFILE=$1
;;
-o | --operation ) shift
OPERATION=$1
;;
-h | --help ) usage
exit 1
;;
* ) usage
exit 2
esac
shift
done
get_secureUPN $ADSYNCURL
uploadFile $CSVFILE $OPERATION