You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-31 - 11:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Wed Dec 31, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
758581215156,91sleep30-21swapper/319:06:443
75851829157,9sleep20-21swapper/219:05:592
75852228259,6sleep00-21swapper/019:06:010
75847827457,10sleep10-21swapper/119:05:301
758963993229,1cyclictest0-21swapper/322:15:003
75896399252,8cyclictest1062449-21turbostat23:54:593
758963992521,2cyclictest0-21swapper/322:35:193
9997492221,9sleep1999961-21cut22:55:171
758963992219,2cyclictest0-21swapper/300:15:273
758963992219,1cyclictest0-21swapper/322:15:153
758963992218,2cyclictest0-21swapper/300:05:003
9141432210,3sleep2421rcuc/221:35:012
758963992118,1cyclictest0-21swapper/323:15:273
75896399211,17cyclictest1051913-21turbostat23:45:003
758963992017,1cyclictest0-21swapper/323:35:153
758963992017,1cyclictest0-21swapper/320:40:153
758963992016,2cyclictest0-21swapper/300:10:003
75896399201,12cyclictest961939-21taskset22:20:113
75896399200,16cyclictest0-21swapper/323:13:453
75896399192,11cyclictest793260-21taskset19:40:153
758963991916,1cyclictest0-21swapper/323:45:233
758963991915,1cyclictest0-21swapper/320:40:013
75896399191,14cyclictest871144-21chrt20:50:203
75896399191,14cyclictest0-21swapper/322:25:123
75896399191,14cyclictest0-21swapper/300:30:173
9341282181,5sleep2934182-21ntpq21:50:212
75896399188,6cyclictest945519-21taskset22:04:243
758963991815,2cyclictest0-21swapper/322:40:183
758963991815,1cyclictest0-21swapper/323:25:003
758963991815,1cyclictest0-21swapper/323:00:003
758963991815,1cyclictest0-21swapper/320:15:243
758963991814,2cyclictest0-21swapper/323:00:123
75896399181,13cyclictest0-21swapper/321:25:223
75896399181,13cyclictest0-21swapper/320:00:123
75896399181,11cyclictest1079355-21taskset00:10:143
75896399181,10cyclictest995458-21taskset22:50:173
75896399181,10cyclictest798881-21taskset19:45:153
9819422171,2sleep00-21swapper/022:35:200
7935302171,10sleep1793742-21hddtemp_smartct19:40:171
758963991714,1cyclictest0-21swapper/321:55:013
758963991714,1cyclictest0-21swapper/321:35:263
758963991713,2cyclictest0-21swapper/321:30:183
758963991712,2cyclictest0-21swapper/319:35:013
75896399171,13cyclictest0-21swapper/300:24:403
75896399171,12cyclictest1099830-21turbostat00:30:003
75896399171,11cyclictest938295-21taskset21:55:183
75896399170,13cyclictest0-21swapper/321:20:203
75896399170,10cyclictest924880-21taskset21:45:123
11040582171,12sleep11104106-21modprobe00:30:241
9606642161,10sleep2960718-21ntpq22:15:222
9513652161,4sleep2951461-21tune2fs22:10:172
9193272161,10sleep0919399-21/usr/sbin/munin21:40:000
7869762161,10sleep0787130-21sed19:30:260
75896399166,8cyclictest924348-21chrt21:41:363
75896399162,10cyclictest819714-21taskset20:05:113
75896399161,9cyclictest972668-21taskset22:30:093
75896399161,9cyclictest771616-21taskset19:20:063
758963991613,1cyclictest0-21swapper/320:25:143
758963991613,1cyclictest0-21swapper/320:20:253
758963991612,2cyclictest0-21swapper/321:15:263
758963991611,3cyclictest0-21swapper/319:50:133
758963991611,3cyclictest0-21swapper/319:15:103
75896399161,10cyclictest889509-21taskset21:10:163
75896399160,13cyclictest0-21swapper/323:05:173
75896399160,13cyclictest0-21swapper/322:45:143
75896399160,13cyclictest0-21swapper/319:35:203
75896399160,12cyclictest0-21swapper/322:05:123
75896399160,10cyclictest1067908-21taskset23:59:113
9553722150,7sleep1955377-21/usr/sbin/munin22:10:281
8348912151,9sleep1835135-21users20:15:291
75896399151,9cyclictest877213-21taskset20:58:073
75896399151,9cyclictest846791-21taskset20:30:163
75896399151,9cyclictest830017-21taskset20:11:443
758963991512,1cyclictest0-21swapper/320:45:303
75896399150,13cyclictest0-21swapper/323:25:123
75896399150,12cyclictest0-21swapper/323:31:173
75896399150,12cyclictest0-21swapper/321:05:223
75896399150,12cyclictest0-21swapper/319:55:213
75896399150,12cyclictest0-21swapper/319:10:183
75896399150,12cyclictest0-21swapper/300:35:033
10106632151,9sleep01012287-21cut23:05:180
8784812141,2sleep00-21swapper/021:00:170
8473172141,8sleep0848946-21cut20:30:180
7865082141,8sleep1786646-21nfs4_client19:30:191
75896399141,10cyclictest882625-21turbostat21:05:003
75896399140,11cyclictest7810882chrt19:25:193
9508192131,8sleep1950846-21dump-pmu-power22:10:001
8686762131,8sleep0870386-21cut20:50:180
8657582133,6sleep0865880-21netstat20:45:210
11095332130,9sleep01109579-21/usr/sbin/munin00:35:210
10521592131,3sleep21052178-21apt-get23:45:122
9411532121,7sleep0941632-21tr22:00:170
9359112121,7sleep1936621-21kthreadcore21:55:171
8989332121,7sleep2899145-21idleruntime21:20:152
8572742121,6sleep0857361-21kthreadcore20:40:160
8564732121,7sleep0856526-21sendmail20:40:000
8299322120,1sleep00-21swapper/020:10:420
7934532120,8sleep0793239-21/usr/sbin/munin19:40:160
10945332121,7sleep21094655-21latency_hist00:25:012
10840392121,7sleep01084250-21latency_hist00:15:000
10153492120,7sleep21015495-21gltestperf23:10:132
10086842120,8sleep21008811-21systemctl23:00:232
9986932111,5sleep0998993-21gpg-connect-age22:55:000
9886232111,5sleep0988746-21diskstats22:45:140
9837822111,6sleep0985065-21cat22:40:170
9257932110,7sleep1927427-21cut21:45:191
8465232111,5sleep1846625-21cut20:30:151
8409802111,4sleep2840999-21apt-get20:25:122
8240352110,1sleep00-21swapper/020:05:260
8058502111,6sleep0807488-21latency19:50:180
7591332112,4sleep2759152-21apt-get19:10:122
10964762111,6sleep01098189-21kthreadcore00:25:180
10843012112,4sleep11084320-21apt-get00:15:131
10778062111,5sleep01077855-21ntpq00:05:220
10723382111,5sleep21072391-21ntpq00:00:242
10349572111,3sleep11031088-21/usr/sbin/munin23:25:191
10205772111,5sleep21020786-21hddtemp_smartct23:15:152
9976882101,5sleep2997735-21ntpq22:50:212
9942062104,2sleep1994277-21irqstats22:50:161
9726702101,4sleep0972689-21apt-get22:30:110
9697862101,4sleep0970971-21memory22:25:190
9671882101,4sleep2967207-21apt-get22:25:122
9653882101,5sleep1965870-21tail22:20:181
9455682101,4sleep2945590-21munin-run22:05:002
9445992101,4sleep1944648-21ntpq22:00:221
8871372100,6sleep0887212-21timerandwakeup21:05:250
8710562101,4sleep2871137-21ntpq20:50:202
7932382101,4sleep2793257-21apt-get19:40:142
7784682101,5sleep0780167-21tr19:25:170
7755422101,4sleep1775591-21ntpq19:20:221
7707172101,4sleep2770961-21users19:15:252
11050252101,5sleep01105165-21gpgv00:35:000
10886782101,4sleep01088880-21users00:15:280
10882772101,4sleep21088320-21ntpq00:15:222
10613262101,4sleep01061420-21ntp_192.168.11423:50:200
10416502101,3sleep00-21swapper/023:35:120
10403992101,5sleep11040449-21ntpq23:30:201
10363992101,5sleep01036514-21grep23:30:120
10295042101,4sleep11029552-21ntpq23:20:181
10190062101,4sleep01019020-21ntpq23:10:190
10148022101,4sleep11014851-21munin-run23:10:001
10088622101,5sleep11009054-21sed23:00:261
985066291,4sleep1986828-21grep22:40:181
951084291,2sleep00-21swapper/022:10:150
946414291,4sleep0947831-21cut22:05:190
934371291,4sleep1934385-21mailstats21:50:241
923643291,4sleep0923690-21ntpq21:40:210
918311291,4sleep1918360-21ntpq21:35:211
906580291,4sleep1907682-21memory21:25:181
898351291,4sleep0898606-21date21:20:000
886790291,4sleep2886840-21ntpq21:05:192
876496290,1sleep10-21swapper/120:55:241
865892291,4sleep1865931-21ntpq20:45:221
853363291,4sleep20-21swapper/220:35:182
852048291,4sleep1853360-21kthreadcore20:35:161
843074291,4sleep0844771-21sed20:25:170
835510291,4sleep0835581-21munin-run20:20:000
831103291,4sleep2832184-21cut20:15:182
819692291,2sleep20-21swapper/220:05:112
803682291,3sleep2803701-21apt-get19:50:122
802419291,4sleep0802468-21ntpq19:45:210
798757294,2sleep2421rcuc/219:45:152
791904291,3sleep0791952-21ntpq19:35:190
787718290,1sleep20-21swapper/219:35:012
1105585291,3sleep11105841-21df00:35:131
1098922291,4sleep11098997-21/usr/sbin/munin00:25:231
1083090291,3sleep21083144-21ntpq00:10:222
1073839291,3sleep21073858-21apt-get00:05:122
1031376291,4sleep01031523-21grep23:25:140
988957280,4sleep20-21swapper/222:45:172
988037280,1sleep20-21swapper/222:42:592
982888280,1sleep20-21swapper/222:39:362
982231281,3sleep1982326-21timerandwakeup22:35:261
957189281,3sleep1957920-21kthreadcore22:15:171
950581281,3sleep2950582-21sshd22:07:002
936623281,3sleep2938294-21kthreadcore21:55:182
908841281,3sleep2908981-21ls21:30:002
872413280,3sleep2872432-21apt-get20:55:132
836243280,4sleep20-21swapper/220:20:142
819991281,4sleep1820206-21basename20:05:151
809881281,3sleep0811514-21kthreadcore19:55:200
786918280,1sleep20-21swapper/219:30:252
775593280,4sleep2775638-21ntpq19:20:232
766965281,4sleep0768452-21cut19:15:150
1105842281,3sleep21105912-21/usr/sbin/munin00:35:142
1089556281,3sleep01089576-21cpu00:20:130
1078114280,1sleep10-21swapper/100:05:271
1068284281,2sleep00-21swapper/000:00:130
1062996281,3sleep21063168-21cstates23:55:142
1058883280,4sleep21060595-21grep23:50:172
1041925281,3sleep21041651-21/usr/sbin/munin23:35:142
1020860281,3sleep00-21swapper/023:15:160
992559271,3sleep1992646-21processes22:45:241
972692271,2sleep10-21swapper/122:30:121
961917271,2sleep2961936-21apt-get22:20:112
930106271,2sleep00-21swapper/021:50:110
924882271,1sleep00-21swapper/021:45:140
924528270,1sleep10-21swapper/121:43:501
923459271,3sleep2923598-21awk21:40:202
916147270,3sleep2917899-21kthreadcore21:35:182
907870271,3sleep0907918-21ntpq21:25:210
898020270,1sleep10-21swapper/121:15:531
898011270,1sleep20-21swapper/221:15:462
887759270,1sleep10-21swapper/121:09:071
882509270,1sleep20-21swapper/221:03:552
882281270,1sleep10-21swapper/121:01:021
871905271,4sleep10-21swapper/120:51:521
866696270,1sleep20-21swapper/220:47:122
839169270,3sleep1839666-21strings20:20:171
837428271,3sleep00-21swapper/020:20:160
818968270,1sleep10-21swapper/120:02:381
808451270,1sleep10-21swapper/119:52:071
792664270,1sleep10-21swapper/119:36:241
771623270,1sleep00-21swapper/019:20:120
771120270,1sleep10-21swapper/119:17:121
763267271,3sleep1331ktimers/119:10:241
1094795271,1sleep20-21swapper/200:25:122
1062940270,3sleep11062448-21apt-get23:55:011
1057033270,1sleep00-21swapper/023:48:270
1051753271,4sleep00-21swapper/023:43:110
1004121270,1sleep00-21swapper/022:58:170
976981260,3sleep20-21swapper/222:30:252
971915260,1sleep10-21swapper/122:27:161
963663261,2sleep00-21swapper/022:20:170
961477260,1sleep00-21swapper/022:18:010
940014260,1sleep00-21swapper/021:56:080
913847260,1sleep10-21swapper/121:32:251
913363261,2sleep03468576-21kworker/0:1+events21:30:270
902939260,3sleep10-21swapper/121:20:261
902653260,2sleep00-21swapper/021:20:200
892869260,4sleep10-21swapper/121:12:131
892400260,3sleep00-21swapper/021:10:270
888661260,2sleep20-21swapper/221:10:162
877303260,1sleep00-21swapper/020:59:160
857489261,2sleep10-21swapper/120:40:171
850937260,1sleep20-21swapper/220:31:332
845322261,2sleep10-21swapper/120:25:271
830259260,4sleep10-21swapper/120:14:521
825829260,2sleep20-21swapper/220:10:162
818846260,1sleep00-21swapper/020:01:040
808927260,1sleep10-21swapper/119:55:041
803115260,1sleep10-21swapper/119:46:101
782013260,1sleep10-21swapper/119:28:551
764049260,4sleep00-21swapper/019:13:550
1094452262,2sleep1758958-21cyclictest00:23:591
1079636261,2sleep10-21swapper/100:10:171
1069142261,2sleep10-21swapper/100:00:191
1067710260,1sleep00-21swapper/023:56:400
1057985261,2sleep10-21swapper/123:50:171
1047198261,1sleep10-21swapper/123:40:151
1046902261,1sleep20-21swapper/223:40:112
1046300260,4sleep10-21swapper/123:36:011
1035094260,2sleep20-21swapper/223:25:202
1030551261,2sleep20-21swapper/223:25:002
1025030260,1sleep10-21swapper/123:16:531
1015133261,2sleep10-21swapper/123:10:121
1014469260,4sleep20-21swapper/223:05:492
999199250,1sleep20-21swapper/222:55:032
861015250,2sleep20-21swapper/220:40:262
814444250,1sleep20-21swapper/220:00:142
780969250,1sleep20-21swapper/219:25:192
1105010251,2sleep2758958-21cyclictest00:34:502
1040697250,1sleep20-21swapper/223:30:252
1029104250,1sleep00-21swapper/023:20:180
924879240,1sleep20-21swapper/221:45:122
808938240,1sleep20-21swapper/219:55:132
1052450240,1sleep10-21swapper/123:45:151
1004386240,1sleep00-21swapper/023:00:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional