You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 20:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Thu Nov 27, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7213069911980,1194cyclictest7616622sleep307:40:003
7213069911650,1161cyclictest9155042sleep309:34:573
721306993220,319cyclictest825083-21turbostat08:34:563
721306992319,1cyclictest0-21swapper/310:04:593
721306992118,1cyclictest0-21swapper/310:50:003
721306992118,1cyclictest0-21swapper/308:25:153
721306991815,1cyclictest0-21swapper/309:50:013
190898180,4rtkit-daemon0-21swapper/211:27:392
721306991714,1cyclictest0-21swapper/310:15:003
721306991714,1cyclictest0-21swapper/308:55:143
72130699171,12cyclictest1102018-21turbostat11:30:013
190898170,4rtkit-daemon1172747-21ssh12:13:430
9331212160,4sleep0934264-21tr09:45:170
72130699169,3cyclictest1165226-21turbostat12:10:003
72130699167,6cyclictest7394852sleep307:20:193
72130699167,6cyclictest10521722sleep310:57:043
721306991613,1cyclictest0-21swapper/312:39:123
721306991613,1cyclictest0-21swapper/311:55:013
72130699161,12cyclictest1083531-21chrt11:15:223
72130699161,12cyclictest1021193-21taskset10:40:013
72130699161,11cyclictest964564-21taskset10:05:143
72130699155,7cyclictest1171473-21taskset12:10:483
721306991513,1cyclictest0-21swapper/311:30:393
721306991513,0cyclictest0-21swapper/308:51:503
721306991512,1cyclictest0-21swapper/312:30:073
721306991512,1cyclictest0-21swapper/311:36:433
721306991512,1cyclictest0-21swapper/311:13:463
721306991512,1cyclictest0-21swapper/311:04:053
721306991512,1cyclictest0-21swapper/311:04:043
721306991512,1cyclictest0-21swapper/308:05:013
721306991512,1cyclictest0-21swapper/307:36:183
721306991512,1cyclictest0-21swapper/307:15:143
72130699151,11cyclictest980887-21taskset10:15:153
72130699151,11cyclictest868127-21taskset09:00:193
72130699151,11cyclictest804577-21taskset08:10:173
72130699151,11cyclictest1178924-21taskset12:15:223
8626602141,10sleep0862710-21sendmail09:00:010
72130699145,7cyclictest1131762-21taskset11:47:203
72130699145,6cyclictest7820022sleep307:55:163
72130699144,7cyclictest921567-21taskset09:35:203
72130699144,7cyclictest9072702sleep309:29:363
72130699144,7cyclictest1043454-21taskset10:50:513
72130699144,7cyclictest10041052sleep310:28:273
721306991413,0cyclictest0-21swapper/312:00:363
721306991413,0cyclictest0-21swapper/307:31:023
721306991412,1cyclictest0-21swapper/311:41:283
721306991412,1cyclictest0-21swapper/309:15:443
721306991412,0cyclictest0-21swapper/311:09:313
721306991411,1cyclictest0-21swapper/310:31:563
721306991411,1cyclictest0-21swapper/308:45:583
72130699141,10cyclictest948206-21taskset09:55:093
72130699141,10cyclictest931710-21turbostat09:45:013
72130699141,10cyclictest873719-21taskset09:05:163
72130699141,10cyclictest811887-21taskset08:16:243
72130699140,11cyclictest843487-21taskset08:40:243
72130699140,11cyclictest843487-21taskset08:40:233
72130699140,11cyclictest767940-21chrt07:45:013
72130699140,11cyclictest747839-21chrt07:25:193
72130699140,11cyclictest1141172-21taskset11:55:023
190898140,9rtkit-daemon1085450-21ssh11:19:170
10866552141,10sleep01087648-21irqstats11:20:170
7807342130,4sleep0780849-21perl07:55:140
72130699134,6cyclictest11959962sleep312:25:523
72130699133,7cyclictest837293-21taskset08:36:183
72130699131,9cyclictest891538-21chrt09:20:013
72130699131,9cyclictest881299-21taskset09:10:323
72130699131,9cyclictest818588-21taskset08:22:483
721306991312,0cyclictest0-21swapper/312:21:143
721306991312,0cyclictest0-21swapper/307:14:513
721306991311,1cyclictest0-21swapper/310:20:153
721306991311,1cyclictest0-21swapper/309:52:083
72130699131,10cyclictest1028026-21taskset10:42:323
721306991310,2cyclictest0-21swapper/308:05:233
72130699130,10cyclictest1091195-21taskset11:20:203
190898130,1rtkit-daemon0-21swapper/010:48:420
11172632131,2sleep20-21swapper/211:37:152
7487122120,1sleep0748884-21latency_hist07:30:020
72130699121,8cyclictest7742552sleep307:50:163
190898120,8rtkit-daemon1076281-21ssh11:11:381
9622462111,4sleep0962254-21ssh10:00:360
7207282110,3sleep30-21swapper/307:05:443
7155662111,3sleep2716349-21irqrtprio07:05:192
190898110,4rtkit-daemon1109173-21ssh11:32:510
9897982103,4sleep2818535-21kworker/u16:1+nfsiod10:20:172
9466352100,7sleep1946643-21ssh09:52:061
8253492101,7sleep10-21swapper/108:30:131
190898100,7rtkit-daemon727881-21munin-node09:10:472
190898100,6rtkit-daemon956338-21/usr/sbin/munin10:00:201
190898100,6rtkit-daemon1063382-21ssh11:05:161
190898100,5rtkit-daemon891652-21apt-get09:20:141
190898100,5rtkit-daemon859716-21grep08:55:172
190898100,4rtkit-daemon1149289-21ssh12:00:020
12143132100,6sleep01214494-21sed12:40:010
941679293,3sleep0943631-21kthreadcore09:50:170
913712290,6sleep0913720-21ssh09:30:500
898286290,1sleep00-21swapper/009:22:330
825313291,4sleep0825346-21apt-get08:30:120
819190291,5sleep2819218-21cut08:25:122
763782290,6sleep1765837-21grep07:40:201
735146290,6sleep2735221-21ntp_states07:15:202
19089890,6rtkit-daemon727881-21munin-node09:31:211
19089890,5rtkit-daemon811627-21timerwakeupswit08:15:232
19089890,5rtkit-daemon438-21wpa_supplicant11:00:460
19089890,5rtkit-daemon438-21wpa_supplicant11:00:450
19089890,5rtkit-daemon438-21wpa_supplicant09:05:202
19089890,5rtkit-daemon1928-21sendmail-mta12:25:320
19089890,5rtkit-daemon1141438-21sh11:55:140
19089890,4rtkit-daemon994984-21ssh10:21:361
19089890,4rtkit-daemon900021-21ssh09:25:141
19089890,4rtkit-daemon1163193-21users12:05:230
19089890,4rtkit-daemon1051990-21ssh10:56:460
19089890,4rtkit-daemon1022480-21ssh10:40:152
19089890,3rtkit-daemon667-21snmpd11:44:230
19089890,3rtkit-daemon1132468-21ssh11:49:020
19089890,1rtkit-daemon0-21swapper/208:38:282
1198667290,7sleep11198374-21/usr/sbin/munin12:30:161
1196275290,2sleep2875241-21diskmemload12:26:272
1190489290,6sleep11191164-21irqrtprio12:25:141
1147020291,1sleep1171rcu_preempt11:55:251
1038604291,6sleep00-21swapper/010:50:170
1010367291,5sleep01010399-21/usr/sbin/munin10:30:200
996951281,4sleep2996988-21apt-get10:25:102
988088280,6sleep2988096-21ssh10:19:322
970719281,4sleep2970727-21ssh10:06:302
955361281,4sleep2955365-21ssh09:58:292
954058280,6sleep1954066-21ssh09:55:281
946054280,6sleep2946062-21ssh09:50:472
924392280,5sleep1925386-21irqstats09:40:161
922048281,4sleep0922056-21ssh09:36:270
916041281,4sleep1916179-21grep09:35:111
869231281,3sleep0869264-21apt-get09:05:100
839608280,6sleep0841639-21kthreadcore08:40:180
839608280,6sleep0841639-21kthreadcore08:40:170
774292280,5sleep0774441-21df_inode07:50:170
761398281,4sleep1761419-21munin-run07:39:591
743105281,5sleep2744003-21cut07:25:152
720792281,4sleep0720794-21sshd07:06:330
19089880,5rtkit-daemon667-21snmpd08:10:362
19089880,4rtkit-daemon818534-21chrt08:22:062
19089880,4rtkit-daemon81364-21packagekitd11:35:100
19089880,4rtkit-daemon727881-21munin-node12:16:430
19089880,4rtkit-daemon727881-21munin-node09:43:240
19089880,4rtkit-daemon667-21snmpd10:32:542
19089880,4rtkit-daemon667-21snmpd09:17:491
19089880,4rtkit-daemon667-21snmpd08:50:162
19089880,4rtkit-daemon667-21snmpd08:40:252
19089880,4rtkit-daemon667-21snmpd08:40:252
19089880,4rtkit-daemon667-21snmpd07:48:042
19089880,4rtkit-daemon438-21wpa_supplicant07:41:352
19089880,4rtkit-daemon1928-21sendmail-mta07:10:382
19089880,1rtkit-daemon0-21swapper/208:31:012
1181719280,5sleep01181877-21diskstats12:20:130
1149310280,4sleep11149343-21apt-get12:00:091
1138950281,5sleep21138956-21ssh11:50:262
1103351281,4sleep11104470-21cut11:30:161
1070176281,4sleep21070381-21hddtemp_smartct11:10:142
962227270,4sleep2962235-21ssh10:00:332
934265270,4sleep1936288-21cut09:45:181
930011270,2sleep2875241-21diskmemload09:41:012
910165271,4sleep20-21swapper/209:30:202
857717270,4sleep1858917-21cut08:55:171
848052271,3sleep0849077-21memory08:45:210
825057271,4sleep1825102-21unin-run08:30:001
819011270,3sleep0819044-21apt-get08:25:100
793111271,3sleep1793254-21df08:05:131
786852271,3sleep0786900-21/usr/sbin/munin08:00:100
781009271,3sleep20-21swapper/207:55:152
767953271,3sleep0767985-21apt-get07:45:110
760542270,4sleep0760762-21unixbench_multi07:35:220
721132270,4sleep0721090-21apt-get07:10:020
715378270,4sleep1715564-21idleruntime07:05:181
19089870,4rtkit-daemon727881-21munin-node12:31:020
19089870,4rtkit-daemon727881-21munin-node08:03:082
19089870,4rtkit-daemon674-21lldpd07:20:422
19089870,4rtkit-daemon473-21gmain11:13:540
19089870,3rtkit-daemon883039-21ssh09:14:291
19089870,3rtkit-daemon754035-21strings07:30:182
19089870,3rtkit-daemon727881-21munin-node08:05:392
19089870,3rtkit-daemon671-21ntpd10:13:222
19089870,3rtkit-daemon671-21ntpd08:46:052
19089870,3rtkit-daemon667-21snmpd07:35:232
19089870,3rtkit-daemon1094692-21basename11:25:180
19089870,3rtkit-daemon1061478-21ssh11:04:541
19089870,3rtkit-daemon1061478-21ssh11:04:531
19089870,3rtkit-daemon1020309-21ssh10:38:282
19089870,3rtkit-daemon1002229-21ntp_states10:25:191
1214043270,5sleep21214049-21ssh12:39:182
1156728271,4sleep21156735-21sh12:04:072
1138356270,1sleep00-21swapper/011:50:200
1127991270,4sleep11130138-21grep11:45:191
1108611270,5sleep2299-21systemd-journal11:31:332
1093506270,3sleep11093514-21ssh11:24:031
1067325270,5sleep01067528-21/usr/sbin/munin11:05:230
1045579270,3sleep21045618-21apt-get10:55:112
1008334270,4sleep11010139-21sed10:30:171
985698260,1sleep00-21swapper/010:15:220
972570261,2sleep10-21swapper/110:10:151
970905260,1sleep00-21swapper/010:07:000
970887260,1sleep10-21swapper/110:06:571
921876260,1sleep20-21swapper/209:36:042
905569260,4sleep2905577-21ssh09:25:342
898794260,1sleep20-21swapper/209:23:472
890742260,4sleep0890750-21ssh09:18:470
885497260,4sleep2887491-21kthreadcore09:15:182
853047260,3sleep1855196-21grep08:50:201
831757260,3sleep0831928-21gltestperf08:35:130
793075261,2sleep0793108-21apt-get08:05:120
761715260,4sleep0761810-21cpuspeed_turbos07:40:150
742410260,1sleep10-21swapper/107:25:021
19089860,3rtkit-daemon667-21snmpd10:45:012
19089860,3rtkit-daemon473-21gmain07:50:032
19089860,3rtkit-daemon1928-21sendmail-mta09:02:482
1204997260,1sleep20-21swapper/212:32:172
1189053260,4sleep21189061-21ssh12:23:592
1185226260,4sleep11186685-21latency12:20:181
1180302260,1sleep10-21swapper/112:17:381
1173588260,3sleep21173596-21ssh12:15:102
1165854261,3sleep11166022-21cat12:10:151
1164503260,1sleep20-21swapper/212:08:252
1147565260,1sleep20-21swapper/211:56:422
1131309260,1sleep20-21swapper/211:46:242
1112156260,3sleep11113797-21ps11:35:171
1101932260,1sleep10-21swapper/111:29:531
1092065261,3sleep21092071-21ssh11:20:432
1067730260,1sleep20-21swapper/211:05:332
1032612261,3sleep11029412-21/usr/sbin/munin10:45:201
1025196260,3sleep01026423-21memory10:40:190
999064250,2sleep01001055-21kthreadcore10:25:160
954707250,1sleep00-21swapper/009:56:580
938091250,3sleep20-21swapper/209:46:072
899670250,2sleep00-21swapper/009:25:120
868897250,1sleep00-21swapper/009:03:570
867729250,2sleep1868049-21strings09:00:171
849078250,2sleep10-21swapper/108:45:221
843159250,2sleep10-21swapper/108:40:231
843159250,2sleep10-21swapper/108:40:221
812743252,1sleep10-21swapper/108:20:111
807217251,2sleep00-21swapper/008:15:160
801397250,3sleep1803413-21kthreadcore08:10:161
786412250,1sleep10-21swapper/107:58:181
769440250,2sleep10-21swapper/107:45:161
749161250,3sleep1749361-21hddtemp_smartct07:30:141
742162250,1sleep10-21swapper/107:21:481
735924250,1sleep10-21swapper/107:18:121
1213612251,1sleep10-21swapper/112:38:161
1166023251,2sleep21165541-21/usr/sbin/munin12:10:162
1163953250,3sleep10-21swapper/112:07:081
1134714250,2sleep11136082-21cut11:50:171
1084993250,3sleep11085001-21ssh11:18:111
1078221250,2sleep20-21swapper/211:15:142
1059543250,1sleep20-21swapper/211:00:292
1059543250,1sleep20-21swapper/211:00:292
1043566250,3sleep10-21swapper/110:51:061
1042914250,3sleep21043100-21sed10:50:232
1013219250,3sleep01013380-21diskstats10:35:130
1013158250,2sleep10-21swapper/110:35:111
980633241,1sleep10-21swapper/110:15:141
831929241,1sleep10-21swapper/108:35:131
812611240,1sleep00-21swapper/008:20:020
805522240,3sleep00-21swapper/008:13:520
788495240,2sleep10-21swapper/108:00:151
735823241,1sleep00-21swapper/007:16:560
727369240,3sleep10-21swapper/107:11:001
1124931241,1sleep20-21swapper/211:44:332
1118737240,1sleep10-21swapper/111:40:031
1052093240,1sleep11052101-21ssh10:56:541
1021750240,2sleep11022278-21irqcore10:40:161
992973230,1sleep00-21swapper/010:20:180
972499230,1sleep00-21swapper/010:10:050
875719230,1sleep00-21swapper/009:10:140
869267230,1sleep10-21swapper/109:05:111
850403230,1sleep00-21swapper/008:50:140
806114230,1sleep10-21swapper/108:15:021
780139230,1sleep10-21swapper/107:53:031
749005230,1sleep00-21swapper/007:30:130
736339230,1sleep00-21swapper/007:20:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional